DATA DUPLICATION VILLAGE - Facts, figures, fun from managing 100,000 hard drives
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Alternative Title |
| |
| Title of Series | ||
| Number of Parts | 322 | |
| Author | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/39879 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
DEF CON 26101 / 322
18
27
28
40
130
134
164
173
177
178
184
190
192
202
203
218
219
224
231
233
234
235
237
249
252
255
268
274
287
289
290
295
297
298
299
302
306
309
312
315
316
00:00
Integrated development environmentData storage deviceSystem administratorTraffic reportingPhysical systemMereology1 (number)Remote procedure callCodeDependent and independent variablesMultiplication signBitEnterprise architecturePoint (geometry)StatisticsData storage deviceIntegrated development environmentHard disk driveComputer fileData centerBasis <Mathematik>PrototypeRight angleNumberData conversionView (database)Group actionComputer animation
04:00
Data storage deviceProcess (computing)Physical systemGraph coloringBitRight angleInternetworkingData centerOperator (mathematics)
05:42
Open setSource codeNP-hardHard disk drivePlastikkarteMetropolitan area networkStatisticsMultiplicationMultiplication signSet (mathematics)NumberSequelUniform resource locatorVideo gameWebsiteComputer animation
07:02
RAIDError messageSurfaceAttribute grammarStatisticsPlastikkarteSet (mathematics)StatisticsChannel capacityDreizehnBusiness modelPlastikkarteRaw image format
08:17
Hill differential equationError messageAttribute grammarPlastikkarteStatisticsDecimalRight angleHuman migrationCASE <Informatik>Physical systemMultiplication signView (database)Row (database)Point (geometry)BitMereologyComputer animation
09:15
Attribute grammarStatisticsPlastikkarteSynchronizationRAIDNumberFrequencyBit rateMiniDiscSurvival analysisElectric currentMeasurementBusiness modelAsynchronous Transfer ModeInclusion mapSoftware maintenanceHuman migrationMereologySoftwareNumberMultiplication signRight angleProcess (computing)Different (Kate Ryan album)BitOperator (mathematics)1 (number)Bit ratePoint (geometry)Slide rulePhysical systemFrequencyRAIDWell-formed formulaTotal S.A.Vector spacePlastikkarteMeasurementStatisticsBusiness modelLevel (video gaming)Error messageFile formatSoftware testingValidity (statistics)NeuroinformatikCountingService (economics)Goodness of fitEntire functionCuboidTraffic reportingCASE <Informatik>ConsistencyComputer animation
17:00
Goodness of fitMachine visionStatisticsProcess (computing)Point (geometry)NumberDescriptive statisticsBusiness modelVideo gameBit rateComputer animation
17:48
NumberPoint (geometry)Bit rateData centerCuboidHard disk driveGoodness of fitComputer animation
18:47
StatisticsBusiness modelBit rateStatisticsMathematicsNumberEntire functionMultiplication signFrequencyFacebookRight angleInformationComputer animation
19:45
Graph (mathematics)Business modelBit rateElectric currentDifferent (Kate Ryan album)Right angleNumberMultiplication signData centerIntegrated development environmentCASE <Informatik>Set (mathematics)Video gameSoftware testingMathematicsBit rateCurveBusiness modelLevel (video gaming)MereologyAreaData storage devicePoint (geometry)MUDException handlingLebensdauerverteilungDebuggerLogic gateBlogDigitizingFitness functionFrequencyComputer animation
25:38
ModemStatisticsBusiness modelVideo gameStatisticsRight angleVideo gameOperator (mathematics)1 (number)Business modelType theoryLogic gateSolid geometryData centerSequence
26:26
StatisticsBusiness modelVideo gameModemElectric currentData storage deviceResultant1 (number)10 (number)
27:14
Business modelStatisticsModemEnterprise architectureMiniDiscPower (physics)1 (number)2 (number)MathematicsPoint (geometry)Slide ruleNeuroinformatikMultiplication signRight angleMiniDiscPopulation densityGoodness of fitPower (physics)Enterprise architectureDigitizingIntegrated development environmentData storage deviceBit rateSingle-precision floating-point formatCuboidInformationNumberPhysical systemGraph (mathematics)MereologyStructural loadDisk read-and-write headCalculationNormal (geometry)Closed setNP-hardComputer animation
34:05
MiniDiscPower (physics)CountingDrum memoryEnterprise architectureSummierbarkeitUnicodeRaw image formatIntrusion detection systemDivisorPairwise comparisonRight angleIntegrated development environmentPower (physics)CuboidStatisticsDecision theoryGreatest elementBit rateLevel (video gaming)PlastikkartePoint (geometry)Goodness of fitEnterprise architectureMultiplication sign1 (number)Data centerAxiom of choiceNumberRaw image formatDifferent (Kate Ryan album)Field (computer science)MathematicsSoftware8 (number)Pairwise comparisonState observerPhysical systemHidden Markov modelError messageFirmwareFrequencyBusiness modelMarginal distributionOcean currentComputer animation
40:24
Bit rateAnnulus (mathematics)MiniDiscSinc functionGoodness of fitOperator (mathematics)BitRange (statistics)FamilyCASE <Informatik>Revision controlStatisticsNumberCross-correlationRAIDMereologyNormal (geometry)Degree (graph theory)Mathematical analysisBuildingSocial classMultiplication sign1 (number)Point cloudAverageBit rateAttribute grammar6 (number)Business model8 (number)Different (Kate Ryan album)Right angleField (computer science)Event horizonData centerHard disk driveSet (mathematics)NeuroinformatikBlock (periodic table)Computer clusterLevel (video gaming)File systemDefault (computer science)40 (number)Process (computing)Data storage deviceGreatest elementCodePlastikkarteCodeBackupDisk read-and-write headEnterprise architectureDataflowReading (process)Standard deviationParameter (computer programming)Latent heatCalculationFitness functionSoftware testingScaling (geometry)Physical systemPoint (geometry)LeakObservational studyMathematicsCore dumpGodState observerDiagram
Transcript: English(auto-generated)
00:00
Thank you all for coming in. I appreciate it standing remotely. This is awesome. We're gonna have to have a bigger venue next year. And everybody's having conversations, if you could keep it down for our talk, please. Thank you. I have the pleasure of introducing the talks at the Data Duplication Village. This is the first time we've had talks this year, and I love seeing your response. Thank you all for coming out. And I also have the pleasure of
00:22
introducing this individual, Andy Klein from Backblaze. We have been paying close attention to the reports that he has been generating for the past several years. Four years now? Four years on a quarterly basis, he's been generating reports to a hard drive and putting them out for society for the community to be able to take advantage of, especially us, so we can
00:41
see what happens in the way of failures and drives to look for. So without further ado, Andy, thank you very much. You're welcome. Thank you, Scott. Much appreciated. All right. So they did let me in here with a title of marketing. So I don't know how that happened. I snuck in under the radar. I actually do a long time ago, I actually coded for a
01:04
living. I played systems administrator for a while and all of that. Then I crossed over to the dark side and became a marketing person. The good part is I'm hard to bluff from a technical point of view, but I still am a marketing person. So take everything I say with a
01:21
grain of salt. All right. So a little bit about I have to do this. We're going to talk today a little bit about our environment, so all about drives and stuff like that, right? How we measure a failure, all right, because we do that. I'll walk you through some of the stats we have and then we'll do some fun stuff like look at enterprise drive
01:43
versus consumer drives and helium drives versus airfield drives and a little bit about the idea of can you actually predict failure on a hard drive? All right. If you have enough statistics and enough numbers, you can predict anything. We'll finish up with temperature.
02:02
Just so you know, for those of you who know Backblaze, this was the original storage pod we built, okay? For those in the back, that's plywood, okay? That's how we prototyped the first one. You can see it up on the rack over there in with that lovely
02:21
little piece of Dell equipment, which cost nine times as much, by the way. So we built that, and then we have actually changed it into those pretty red ones that we'll talk about in a few minutes. So our environment, and it's important that you understand our environment just a little bit because people look at this, the data sometimes, and
02:41
they go that's not how my system is and this is it, and I have two drives and one failed and you guys don't know what you're talking about, okay? This is our environment. It's a data center, 60 drives and a chassis. There are now systems with up to 100 drives in a 4U chassis, right? But that's how we do it. And then we actually
03:03
logically grouped together 20 of them, 20 of those chassis into something called a vault. So when a file comes in, it actually gets sharded across those 20 different chassis, right? So that way we can lose because of the way this is done with our
03:21
own, we created our own encoding, erasure code stuff, mechanism, and we actually open sourced it, by the way, so go to GitHub, you can look it up if you want to steal it. Excuse me, borrow it. Excuse me, make it better. It's out there for you guys to look at. But it's a 17-3 encoding mechanism. So you can lose up to three whole systems.
03:51
Before you lose anything, any data. And long before that ever happens, okay, we're way ahead of that, okay? But drives fail. And this is the kind of mechanism you have to
04:05
build when you want to scale the system, right? And because drives do fail. And we'll see a little bit later how many. Right now we're storing about 600 petabytes of data, so just slightly more than came in the last few days. But I am
04:22
amazed. By the way, these guys deserve a hell of a hand, okay? If they do an amazing job to get all of this stuff replicated for you guys, for anyone who's doing that, so I think that's the first thing to do. Let's give them a hand, okay? Because it is
04:40
hard to maintain all of this, and then they have the little failed drives over there to keep you track. It's kind of fun. We have more than 100,000 drives that are in operation right now. So we stick them in those wonderful red chassis, right? And that's us, okay? And you've probably seen this shot on the Internet because
05:00
everybody loves to say, look at a data center, and it's all pretty and red, right? Little story. The reason they're red was because when we did the very first one and built it, the guy called us and said, what color would you like them? Because they were just metal. We said, I don't know. And he said, I have some red. Literally it's that
05:21
simple. And we said, okay. There was no guy down there with a thing going, hey, give me PMS color 127. No, it didn't happen. He had red. So we stuck with it and it's worked out really well. It's pretty cool. Collecting hard drive data. So we use the
05:46
smart mon tools package. Many of you have heard of that. The data we collect is the smart stats off of that. I'll show you exactly what we get in a few minutes. We collect it once a day. We actually scan the drives multiple times a day. And there's a reason
06:06
we're scanning. We're looking for things that are going wrong. But we keep a copy of the data for each drive for each day. We've been doing that since April of 2013. So that's the data set that's out there. If you would like, the data is public. We
06:23
publish it. It's on our website. There's the URL there for you. You can download it. I explained to you how it's laid out. How you can go in and look at it. And we even give
06:42
you some SQL files to go and play with if you want to go do that. To go test and do your own thing. So if you're ever curious and you've got nothing to do for a couple of days, you can download some of that stuff. Now, what do we have here? I thought I
07:05
wanted to show you. This is for every drive once a day. Capacity, you can see that. And then we carry the smart stats. Smart stats carry a normalized and a raw value for each of the statistics that are out there. There are currently 255 sets you could get. Not
07:28
all of them are used. As a matter of fact, roughly only half are used that we're aware of from the drives that we have. And we collect all of that and we store all of that. So if you want to know what, you know, on Thursday, June 13th, 2017, what the
07:47
smart raw value was, smart 2 raw value was for that drive, we have that. It's in that data set. Okay? I don't know why you would want to know that. But, yeah. And on.
08:21
Okay. I think that's a little bit better. I think I'll stay about this far away. All right. Now, the most important part from the point of view of what we're talking about today is the failure button. So what's a, right? And that's what you can see. There's a failure value, all right, in that record. That means that that drive
08:42
failed on that day. Okay? The way we do it, okay, is we actually will scan for drives. And when a drive comes up and it's not there, we go and try to figure out why. It either failed or sometimes, a lot of times what happens is they took the system
09:04
down for some reason, perhaps migration, data migration or whatever the case may be, but we do get failures as well. And so the only thing that gets marked there is a one for failure. A system drive can disappear from the data because, again, it's being migrated. Maybe we took a pod out for maintenance purposes or something like that.
09:24
Okay? So what is a failure? Okay, and this is the part that's really important. The first two are easy, right? Didn't spin up, we can't see it, the rate array won't do anything with it, whatever the case may be,
09:41
right? That third one drives people nuts. I'll be tired, right? Because it's our educated guess from the smart stats values that we see and other things that are happening with that drive. So it failed, it throws FSEC errors. Okay? And we see it happening
10:02
consistently. We pull it out and we'll pull it out and we'll mark it as a failure. We'll also, by the way, before we mark it as a failure, we run it through two levels of testing on the backside, which are basically nothing more than a quick reformat and then a long-term format, which beats the living tar out of it, that we have
10:22
from one of the manufacturers who shall go nameless. And if the drive passes, then it gets put back into service if it passes both of those. So we're really confident that a drive has failed by the time one of those three things has happened to it. And like I said, we don't have to use it.
10:42
Now, for those of you who are familiar with our drive stats, okay, we have been recording the data and this is, the data you're seeing is through the end of June, right? I could go ahead and pull it through today, but I actually pulled it at the beginning of August and it was no real significant difference, so I didn't bother
11:02
updating all of these slides. You can see the kinds of things we've had, 132,000 drives we've had in play, yada, yada, yada, right? The thing that's important is the failure rate. Everybody looks at the failure rate. He looks at the failure rate. Everybody looks at that. They go, that's the number I care about. I don't care about all of these other numbers.
11:24
And there are two basic ways to compute a failure rate. And one of them is wrong. And let's see if you can figure out which one is wrong. Some people do it this way. Number of failures divided by drives times 100, right? Are an easy formula. And given that data I just showed you,
11:43
the failure rate for all of those drives is 5.62%. Some people do it this way. Failure is divided by what's called drive days. Drive days are a count of each day a drive is in operation. Alright? If it is not in operation, i.e. it failed
12:04
and it's gone, it's no longer counted. And that's the way, and you can see the same kind of numbers, right? Drive days divided by failures. But now you get a number that's almost half as much, 2.56%. Which one is right?
12:21
Well, we use the second method. We use drive days. And the reason is for us, method one, that first method, assumes that every drive has been operating the same period of time. And that may be very valid for you in a situation where you have five drives in a rig in a NAS box
12:44
or something like that, and you want to do that computation. That's a very valid computation, right? We don't. We have drives in and out of this system all of the time. So whether it's a failed drive, whether we take a system down for maintenance, whether we put in a new system. When we put in drives
13:00
now, we put in 1200 drives at a time. That's what one of those vaults takes. 1200 drives, it comes spinning up. The lights dim, everything happens, right? So that's why we use that method. Because it accounts for the fact that we have drives in and out of this system all of the time. So if you're ever on our comments
13:23
and all of that kind of stuff, and people are yelling about this, they usually are thinking the first one is, hey, why doesn't that work? And that's why it doesn't work. I want to make sure I stay on time. Now, thank you.
13:44
Just because other people have, in fact, taken all of that data that I talked about and applied other models to it. So we have an annualized failure rate we create, and we'll see that in a minute. For those of you who have ever been in the medical bit
14:01
side of thing, biology side of thing, there's something called Kaplan-Meier, which is basically a how long will something live versus how often will it fail? And Simon Ernie, he's from Sweden, you can see down there where he publishes it. Each
14:21
quarter he updates it. We publish this data. He publishes it. This is pretty boring little chart, but he's got a lot of other ones that are a whole lot more interesting. But basically this is all of the drives we've ever had over time and the chance that they're going to survive after so many days. So one year, two year, three year, four years,
14:42
you're looking at something in the neighborhood of about 88 percent. Okay? So that's the failure rate over time. How long can something be expected to live? You put in a drive today, there's an 88 percent chance it will survive four years. That's the kind of thing that this one does. And he's done this, by the way, for all of the different
15:02
drive models that we have. So it's kind of cool. And the technique isn't very hard if you know how to do it. And a number of other people have done some fun stuff with the data, but this is a really good one. And he does a really nice job of explaining it. Measurements.
15:24
Okay, so when you look, quarterly we publish the drive stats. And like you said, we've been doing this for about four years now. This is everything we got. Here's all of the data. There's no hiding. It's everything. And each quarter we publish two, sometimes three different looks at it. One is a quarterly look. Tell me everything that happened
15:43
in the last quarter only. The other one is a lifetime look. Given all of the drives we currently have running, how have they been doing since we ever put them in? And then the last one is every drive we ever owned, tell me how it did over the entire period, right? So
16:03
you'll see the data. If you go look at the stats and stuff like that, you'll see all that cool stuff. But that's how you look at the data. So a lot of times we make a mistake, by the way. When we publish the report, the first thing is the quarterly numbers. And people have short attention spans. So they look at the quarterly numbers
16:23
and they immediately just say, oh my goodness, so and so's drive only had a zero percent failure rate. Oh my goodness, they're a great drive. Oh well, yeah, but there's only twenty of them, you know. And they've only had a whole total of five hundred
16:42
and thirty-nine days of which they were there. So make sure you pay attention to the data, the quarterly data in particular. We use the quarterly data as just a mechanism like a vector. Is a drive moving up or down in how it's failing over time. And that's a
17:02
good vision point for us to do. I will tell you, pulling these stats and doing all of the magic that we do and stuff like that, that's like a part-time job. It's like another thing I have to do. And so it's we do it and it's great, but it's not like
17:22
that's in my job description anywhere. So, right now, on the other side, okay, you get some really good looking things because you start to get decent numbers of drives and decent number of drive days. So, for example, I know there's some six, I heard there's a couple of six terabyte drives over there. That particular
17:43
model C8, which you cannot get anymore by the way, sorry, that's failure rate, annualized failure rate is point eight seven percent, less than one percent. That's a pretty good number. That's a pretty good number. There's some four terabyte HGST drives that are stellar, point two six percent.
18:04
That's amazing. That's like a hundred drives, a quarter of one failed over the year. That's pretty stellar numbers. Can't get those anymore. I know you can't because
18:22
we bought every single one of them, Western Digital, who owns HGST, had in warehouses everywhere. We sent guys under trucks and in corners looking for the boxes of hard drives and we bought every single one they had because they were great. And then we opened up a data center with them.
18:46
That's what we do. If you know one thing about Backblaze is we are frugal. We started and we continue to use consumer drives, which we'll talk about in a minute. Lifetime stats, remember I said the stats, now I
19:03
have stats and I go back, this is over the entire period of time, right? A little less information, but you can kind of see that's the math we calculated before. That's a fun number. 7437 drives failed since we started keeping stats.
19:21
Anybody else have that many failures? Unless you're from Google or Facebook or one of those guys, you probably, by the way, their number is going to be a whole lot bigger than that. We crow, we have 600 petabytes, yay! Those guys have, I don't know, a hundred times that. It's amazing. Still proud of what we do.
19:42
Anyway, what we did, I did there is I summarized that for all of the different size drives. So if you're going down to the store what might you be thinking about? Now we had a really good run with two terabyte drives.
20:01
Alright, really good. Every single one we got in the place was great. And then there was this thing in 2011, 2012 called the Thailand drive crisis and we had to buy a lot of drives and they were all three terabyte drives during that period. And let's just say that the drives were not as good and we'll leave it at that.
20:26
And but you guys and the numbers are the numbers, right? Now you look at this and you go wow, look at those twelve terabyte drives. They're doing great. Remember we've only had them for a year or so. So if you start to think about how drives fail, they do seem to follow
20:43
a bathtub curve. So there's a little infant mortality at the beginning and then they kind of settle into a really nice low rate for two, three years. Then they start to bump up at about three and a half to four years and the failure rate starts to go up from there.
21:00
And drives seem to follow it. The interesting part about that is the infant mortality rate for us has gone almost flat at the front end of the curve for some of the bigger drives. I don't know if it's because they're making better drives, they're testing them well, whatever the case may be, but we're just not seeing the same level. We're seeing a really interesting, it's
21:24
almost indistinguishable from the middle of the curve now. So yay for the drive manufacturers for that. Let's see, my manufacturer, everybody always asks
21:40
which manufacturer makes the best drive. It depends. HGST, most of the drives we have for them with the exception of one model are four terabytes or less and that was generally before they were acquired.
22:01
They have, so that number kind of fits that model. The early Seagate's not so good. The late Seagate's pretty darn good. And we don't have enough Toshiba drives yet. I do have 1,214 terabyte Toshiba drives in the warehouse
22:20
that are going to be deployed any day, so that'll help some of these numbers. But that's what we have. So you look at the failure rates, you know, everybody wants that answer. Should I buy Seagate's? And I can divide the room in half and this half is the Western Digital half and that half is the Seagate half.
22:44
And you guys can yell and throw each other stuff and everything like that. And then there's a couple of Toshiba guys down in front and nobody pays attention to them anyway. So I'm not going to try to solve that problem for you. The other thing that gets in the way is our environment is our environment.
23:04
It's a data center. We treat these guys really nice. They go in a nice chassis. They get tested. They get put in there. They're air conditioned. We monitor the electricity going through them. It's all filtered and everything like that. I don't know if that's the same environment you have at home.
23:25
I'll just say it. But this is the way the data will get. A little more stuff. Lifetime. So for the drives that are currently in the data center today running, there's ninety-eight thousand of them and you can see the failure rate. A little less
23:43
than two percent. Still forty-three hundred of those drives have failed. And that's what we consider to be the most relevant thing. I can't go back in history. I don't really care about one terabyte drives anymore because we don't have any. We just
24:02
got rid of the last three terabyte drives like two days ago. And the only reason we even had them was because they were in a rack. There was four pods full of them in a rack. And they were in an area where we don't go. The way
24:22
this particular data center is set up, there's like two or three racks that are just all by themselves in a corner somewhere. And they're caged and everything. And we were gonna build out the rest of that at one point and we ran out of electricity. I guess is the best way to put it.
24:42
Who would have known it? It was too expensive. It was actually cheaper to go and have another data center than it was to drag more electricity into that existing one. And this is the kind of fun math you have to do. So, no, I don't want to go
25:02
spend the five hundred thousand dollars to have PG&E drag another hundred megawatts or whatever it was into that data center. Let's just go open one in Phoenix and it worked out. But these happen to be in a rack, one of those racks in the corner. And those drives just never failed for some reason. Neither. They were in a little set there.
25:21
But they finally left. We had a ceremony for them. They will, we'll do a little blog post about them and everything like that. Because that's the kind of goofy stuff we do. So, just all of those models, right, life style stats of operational models.
25:42
Just so you can see, the ones that are really kind of fun, if you stared all the way over at that right hand column, which one's the best? Well, it's the ten terabyte Seagate's all the way down at the bottom, right? Those have been really rock solid
26:02
drives and Seagate doesn't make ten terabyte drives anymore. They skip right to twelve. They made those for like a week, I swear. And we bought them. We bought twelve hundred of them. And we went to go buy some more. Because what we do is we run a sequence of things, right?
26:21
Very typical data center kinds of things you have to think about. We put in twenty. Twenty is what's called the tome. It's one drive in each of those twenty different pods, right? And we see how they perform. And if they work, if they keep up with everything else that's going on and all of that, then we'll say, great.
26:42
Then we'll build a whole storage pod out of them. So, we had another fifty-nine, because there's sixty in there, right? And then we do it again. And then if we're happy with that, we like the results of that, then we'll go out and we'll fill a vault with them at twelve hundred. And that's what we did with the Seagate ones.
27:02
But apparently we took too long, because they decided they weren't gonna make tens anymore. And now they're making twelves and fourteens are coming. I don't believe they have any fourteens yet. So, sorry, can't get those. It looks like you're gonna need really big drives
27:21
next year anyway. So, it's amazing. The other one that does really well is the HGST, about a half a percent. You can see those two four terabyte ones towards the top, which are really good and rock solid. I mentioned the Seagate sixes were pretty good, too.
27:41
The Seagate eight terabyte ones, we'll talk about those in just a second. That's a really nice thing. Because one of them is consumer drive and the other one is an enterprise drive. Ta da! So, I think it's important
28:01
to understand what we care about, because it's probably may or may not be the same as what you care about when you buy a drive. We care about cost. Number one, the rest of them you can gray out. It's almost that much, right? But that second one, as I mentioned, sometimes power is really interesting
28:23
to us. So, for example, when we put in the enterprise eight terabyte Seagate drives, they were almost one and a half times as much power as the consumer drives. And when you're running on the ragged edge of the amount of power you have in a rack, alright, you can't do that.
28:44
Because otherwise you have to, you can't put in ten strip pods in there, because ten times, you can fill a whole rack, right? You can only put in six. And that is, that's not good for density, right? So, power is important, but then Seagate has a really nice capability in there
29:03
called power technology or something like that. It's on the next slide. Which allows you to adjust the amount of power that you're going to give that drive. And we can get it in there. So, cost right now for us, some place around 2.2 cents to 2.25 cents, right? You can actually get a better price on that
29:23
every once in a while. You'll go down and somebody will be having a sale at Costco or wherever, and you'll be able to get it. It'll actually math out to be less than that. But a long time ago we used to buy drives at Costco.
29:41
Now if we show up and say at Costco that we need a thousand drives, they don't let us have them. So, we buy now straight from the manufacturers or close enough. So, that's about what we pay, okay? Some place in there. The other things in there, fits our usage, like I mentioned earlier,
30:03
if we put a drive in now and it just doesn't work, it fails and that happens sometimes. We put a drive in, we put 20 of those drives in there and they just can't keep up. There's something that's not working right in the environment. We don't use it. Why beat our head against the wall, right?
30:24
Failure rates do matter. So, you saw really nice low numbers. We can tolerate anything in a single digit failure rate. Once you start to get above that, you're starting to plate, you're starting to roll the dice really hard on your durability.
30:40
So, anything above a single digit kind of failure rate, the lower the better. Right now we're running at about 1.1, 1.2 percent and that's a really nice number because it keeps the durability going. And the durability is, remember they're started across 20 different things, 17 and 3
31:03
is a mechanism. So, I can lose three whole systems like I mentioned earlier. That's part of the durability that you do the calculation with. But if I have drive failure rates that are 12 percent, I start and rebuild times that are now starting to approach two weeks on some of the large drives.
31:24
All of a sudden the math starts to get funny. So, we like single digits. Warranty, I don't care about warranty, we don't care about warranty. It's almost not worth it for us. I know it's worth it for most of you, but when a drive fails, the time it takes us to go and fill out all of the information, put it in a box,
31:44
send it off, hopefully get it back. And they're going to send you a refurb drive, which I really don't want. Warranty is not interesting. And then the last one for us is speed. The way when we build that array of 20, there are 60 of those
32:03
in a vault. I have no trouble accepting data onto those disks. None at all. The gating item is you can't get me enough data. I mean, it's that simple.
32:25
Yeah, I don't have the slide with me, actually it's probably on my computer somewhere, but we've seen it when we first started, it was around 11 cents, and then over the 10 or so years we've been doing this, it's come down, and you can literally see drives by size do that. They come down and
32:44
it'll start and it'll go down, then the next one will be introduced and it'll be a little higher, and you've got to weed it out until it gets down to where the previous one was, and they're just consistent. The only time that broke was during the drive crisis, and that broke hard.
33:02
Our drive prices through normal channels went up 3x.
33:27
That's a really good point, and the other thing that factors into that is storage density in a given spot. One of the things we've been doing over the last three or four years now is migrating from the smaller drives to larger drives.
33:42
I take out a 4 and I put in a 12. I just got three times as much storage for approximately the same cost. Now, I got a bunch of four terabyte drives, but I got four years out of them. I turn them over and they get recycled. They end up in a pile in China,
34:05
but hopefully not. That's a good point about that, and there is exactly that kind of math we do. Let's compare. This is an 8 terabyte drive.
34:22
By the same manufacturer, two different models, one's a consumer one, one's an enterprise one. About a year ago, we did it. The failure rates in that second column, you can see where they were. Now you can see the current failure rates. That's within the margin of error,
34:42
by the way. So you're sitting there going, hmm, I could spend $129 to buy that 8 terabyte drive or maybe $159. Or I could spend $429 to buy the enterprise drive. I wonder which one I should do if I'm interested about failure.
35:04
For us, it didn't matter. Now, for us also, let's just say the price of those things is approximately the same. You can't do that. I can do that. Because I buy a million dollars worth of drives
35:22
at a time. But that's what we see out of the data. So if I was looking at that and going, what would I do, I might think a consumer drive, because really the failure is about the same. I will tell you a little difference that we've seen, and this is anecdotal. I don't
35:41
I'm never gonna write this down anywhere. The consumer drives seem to have this tolerance for things happening to them inside, like bad sectors that have to be re-mapped around and all of this kind of stuff, right? The enterprise drives don't seem to have that same tolerance.
36:03
When they start to go, they just go. They don't give you a whole lot of notice. It's kind of like, I don't feel good. Goodbye. It's just an observation, and I think it's just because of the situation. I think I'm gonna put a drive in a consumer system, and consumers are, it's not going
36:22
in a data center. It's going in, it's an external drive, and you have it next to your thing, and you drag it around, and you bring it over to Aunt Molly's house, and you drop it on the floor. So they have a lot of tolerance built into them. But if you're making the decision for yourself about what you gotta think about, these are the things that I think you
36:42
might want to consider. The warranties, of course, are different. Typical enterprise one is five. The consumer ones are two. At one point during the drive crisis, they were one year. And if they could have
37:01
gotten it down to like ninety days, they would have done it during that period. Consumer drives are really much less expensive for just off the shelf. Enterprise drives have a lot more features. Power choice technology, that's what I was trying to think of, for example, from Seagate. But they have a lot more things you can tweak in the firmware.
37:22
To make that drive perform that fits into your environment really well. They are absolutely faster to read and write. Absolutely. Again, we don't care because that's not where the bottleneck is. The bottleneck is just getting the data to us. And we have plenty of network for that. So, it's just a matter of waiting. Kind of sitting around
37:44
doing nothing. But on the consumer side, they do use a whole lot less power out of the box. And again, I mentioned do more for your failure. So, what's just right for you? Yeah. So, they seem to do
38:06
quite well. And like I said, and then they get sick and die. And it's that fast. It's like, it's, sometimes it's hours. You'll see the first little, you know, throw an absolute check or something
38:23
like that. And then it goes offline. You know, two hours later. And you go, I didn't even have time to look at it. You know, they're just, and the HGST by the way seemed to be the same way in its behavior. We just thought it was an HGST versus Seagate thing. But it seems to be the same, it seems to be an enterprise
38:42
versus consumer thing. Alright, helium. We'll have plenty of time for questions and getting all of this stuff out of the way. So, helium. So, any large drive now, starting at about the eight terabyte drives, although there were some six terabyte helium drives, but starting with
39:00
about the eights and moving on up are going to have, are going to be filled with helium. They finally got that technology right. It was they were trying for years to figure out how to keep the helium in there. Because it keeps wanting to get out. And they finally figured it out. They even created a smart stat, the smart stats now to measure the amount of helium. So, for example
39:24
HGST smart stat is 22. And it's 100 is a raw value, and anything less than that means it's leaking. And they have a tolerance number but they haven't told us what it is. But we have a handful of them running in the 90s right now. So, we're trying to figure it out.
39:44
The Toshiba drives we just got in, the 14 terabyte Toshiba drives are helium filled drives. And they have two numbers, 23 and 24. And they measure helium at two different levels inside the drive. A high, they call it high and low. I think it's above and below the ladders, the platters. And it's the same kind of thing.
40:05
They, it's the same kind of thing. They're actually in many ways still learning what that number means to them. A little bit. Because it is a fairly new thing for them. So, what can I tell you about that? So, we have some helium field drives on the top. We have some non-helium field drives on the bottom.
40:24
One of the funny things about the bottom, and we'll talk about temperature in a minute, the 8 terabyte airfield drives ran hot. They did. They ran 3, 4, 5 degrees Celsius hotter than the lower end drives. And it's just
40:43
there's so much going on in there. The helium field drives run a little cooler. They run back at normal levels. And that makes sense. That's one of the things they talked about. And right now, okay, we don't see any difference in the annualized failure rates between helium and air.
41:04
Which actually bodes well. That means they picked a good technology. They moved forward. It didn't cost them anything. You can see the different failure rates out there and you can compare it. Now, if this were gonna be a perfect test, the drive days there would be roughly the same and they're not
41:23
right now. So, it's not quite apples to apples, but it's pretty good. And there's enough data there to start to actually get to that kind of conclusion that it looks like the helium drives are gonna have a reasonably, are gonna be able to perform at least as similarly to airfield drives. They still cost a little bit more
41:42
or a lot in some cases. When we bought those HGST ones there, we bought those like 4 years ago, 3 years ago, about 3 years ago. And those were about $450 a piece. And that was a crazy number for us. We bought 45
42:03
of them. So, it was the most expensive storage pod we'd ever built. But they're doing ok. They're in 3 years now, 3 plus years now. That's a really good annualized failure rate after 3 years. And they're hanging on. So, we'll see. We'll see what's gonna happen. We're gonna track them over time
42:23
and see if over time the helium drives continue to maintain you know, that kind of performance, that kind of failure rates with that. Let's see, what else? Yeah, so we'll continue to do that. Those are the two 8 terabytes. We have, I was gonna throw the 6's in there
42:43
too, but they're really, like I said, there were a couple of 6 terabyte helium field drive models, but they didn't make them in any quantity and they kind of experimented with them. They really started to do it in the 8's is where technology. And if you buy anything above that now, that's what's gonna be in it.
43:03
Chances are it's gonna have helium in it. Interesting little thing though, both of those models up there for the helium are enterprise drives or enterprise class drives. It'll be interesting to see if manufacturers continue to build
43:22
large quantities of consumer drives in that size. The reason we buy the enterprise one is, again, the price is about the same in quantity. Ok, so we buy a bunch of them. I don't know if I could buy, let's say, 12,000
43:42
consumer drives right now. You know, if I wanted to buy 12 terabyte Seagate consumer drives, I don't know if I could buy them. I don't know if anybody would sell them to me. Got it. So, and that's part of the way that drive manufacturers manage their channel.
44:01
You might be able to go to Costco and buy one or two. We did that during the drive, the Thailand drive crisis. We went to Costco and Best Buy and bought drives off the shelf because we couldn't get them anywhere else. But I don't sell them. Well, I was going to say there's not enough Costcos, but there probably is.
44:26
Alright, so enough about that. Smart stats. Can you actually predict if a drive is going to fail or not? So, we tracked five stats by default. We've been doing this for years. We talked to drive manufacturers and lots of folks and they said, hey, these are
44:43
five good ones. And so we tracked these numbers. And one of the things I did a few, a little bit ago, was say, alright, if there's an operational drive, if I looked at all of the drives that are running right now, how many of them have one or more of those attributes
45:04
better, greater than zero? And it's either zero or a number better than zero. Zero is good, anything else is bad. And it just so happened about 4.2% of them were like that. So then I looked at all of the failed drives and I said, well, then how many of the failed drives had the same thing, right? 76.7. So,
45:23
if you're a stats guy, you're looking at that going, that doesn't feel like a very good predictor, right? Those five little stats. It kind of looks like it. It's obvious there's a really big gap in there, but you're not sure. You'd like to see that number on the other side be
45:40
what, 95, right? A couple of deviations out, you start to feel good about it. So, some really smart people, okay, not me, over at IBM, Switzerland, got together and did a wonderful little paper a couple of years ago. And that's where you can find it if you're and if you, if you're not good with math,
46:04
it's a fun read. But what they were able to do by drive model, this was the amazing part, right? By drive model, show that you could actually predict with that kind of certainty, okay, what a drive was gonna fail. And that's pretty amazing if you start to do, think about it, right? So, wouldn't you
46:24
like to know what the ninety-seven percent degree certainty three days ahead of time that a drive was gonna fail? That's awesome, right? Now, you have to calculate that for that particular drive as a CH4 terabyte drive, right?
46:41
Then you gotta calculate that for every single drive. And it gets one of these interesting little things of, well, somebody's gotta have a lot of drives to produce the data to calculate it so that everybody else can use it, okay? So, but it's interesting that the drive stats that they, and that's exactly how
47:01
they did this, they looked at all kinds of drive stats that we had. They used our data to do this with. And you are able to get pretty good. Now, you look at the HGST, not so good. That particular model, three days out, eighty-four percent, I don't know if you really want to throw away
47:20
sixteen percent of your drives that are good, right? I don't, I think the other one is pretty cool. I could deal with three percent. So, it seems like there's some way you can calculate this, right? I, like I said, this is it's not my day job to do all of this stuff. So, we're trying to
47:43
run a backup company and cloud storage company. So, if any of you guys want to do it, we'll give you, the data's there to do those kinds of things. But it is interesting. Now, I've heard people, by the way, talk about drive stats, smart stats, and say it's a bunch of garbage.
48:00
It's a bunch of garbage. The different manufacturers, they just spit out numbers, who cares, right? I don't know, that doesn't look like a bunch of garbage. That looks pretty good. And they were really, if you look through the paper that they did, they really spent a lot of time with it.
48:21
Last things we'll ask, temperature, right, just for the fun of it, because everybody asked this question. A number of years ago, Google did a study, right, and said, eh, temperature doesn't matter. You can just crank up the heat. Turn down the air conditioning, you can go. So, we wanted to figure out if that was true, because I don't mind saving on air conditioning.
48:41
Especially since we built the data center in Phoenix. So, the average temperature of operational drives for us is, you can see, around seventy-seven degrees or so. I converted to Fahrenheit, because we're in America. And you can kind of see how that
49:00
chart is. And there's a handful of them that run at forty-five degrees, but, you know, which is pretty warm by the way. You can really start to see. But all of that is within the range of a drive. Right, and this is taken by the way, right off of Smart. It's the sensor inside
49:20
the drive. So, this isn't like in the chassis, or us sticking a thermometer on top of the thing or anything. This is inside the drive itself. And so, but all of those fit within the parameters that they give you of the operational range of a hard drive. So, we've never had a drive fall outside of those parameters.
49:41
Now, the interesting part would be failure. Is there any correlation to failure? So, I broke it down by drive model. Once again, everybody cares about that. And you can see, the three manufacturers aren't even close, right, how they fail. You know, HGST down at the bottom doesn't look like there's any real correlation. It could be anywhere
50:04
along there. The Seagate one, maybe, but not much. And the Western Digital one is a Batman cow. So, that's what I see. I don't know what you guys see. The front thing is, is towards the end, which is where Google
50:23
spent their time talking about, once you start to get above forty degrees Celsius, you actually do see that bump up of drive failure, right. But we just don't see enough drives fail there to actually say that's what happens, right. They seem to fail in other places. It'll take a while, for example, the Western Digital one at
50:43
thirty degrees Celsius is eighteen percent of their drives are failing there, right. That's not much above their normal temperature. So, I don't know, I don't think there's any real correlation between the failure and during a normal range of operation once you get up there. So, you know
51:02
what we talked about, so I'll leave it with questions since we've got a few minutes here for questions. Anybody got anything? No, no. Oh, sorry. Have we done any
51:22
analysis for what file systems and how they might affect drives and so on, right? The answer is no. We use our file system now I'm going to remember what it is, but it's a standard one. I can't remember what it is, but it's a standard one you, everybody and their brother would use. So, we didn't invent our own or anything like that.
51:44
I do know some folks who have tried that because they do some really funky things as it relates to writing blocks and so on, but we just, the drive, that works with the drive and it works. So, no difference with file systems that I'm aware of. So, anything else? Yeah, so
52:21
the question is, since we started putting our numbers out, have we noticed that the consumer drives have gotten better, have failed less? Yes, we've noticed. While I would like to say I'd like to take credit for that, because sometimes
52:41
transparency is a very good thing, I don't think Seagate is sitting around in boardrooms going, gosh, they just published their data, we better get better. I also think, by the way, they learned a lot, the Thailand Drive crisis was really an awful event for a lot of reasons.
53:02
And they really got hurt during that for a lot of different things, publicity, everything like that. So, I think what you're really seeing over the last few years is just them making drives now with a reliable set of parts and so on.
53:21
I'd like to believe we had some influence on them making better drives. We have good relationships with Seagate, for example, with all the drive manufacturers, maybe they just gave us the good ones, I don't know. But we have observed that, yeah, that's a fair observation. Anything else?
53:41
Yes. How do they keep helium in the helium drives? That's a really good question. I know they spent a lot of time creating the case that goes around it and how they pack it. I don't know the mechanics. I was reading
54:02
an article a few weeks ago about it, about how they did it because we got the Toshiba drives and the history and how it was actually, I think, Western Digital which got the first commercial versions of them out. But I don't know the mechanics and I don't think they share a lot of that. There's this general notion of, hey, we did it and
54:22
we used the Flexicor LoaBaw marketing name thing to do it, but they don't give you the specifics of hey, we coded it here and we did this here and we plugged this gap here and all of that. I mean, it can't let anything out. Helium is just going to leak out. It could actually leak out through a lot
54:42
of substances. So that's right. There is no flow. So they've had to reinvent the drive
55:01
a little bit. The question was is with air drives, air helps the heads a little bit there and helium doesn't do that because there is no flow basically. How do you get rid of heat? So those are the kinds of things that they have managed to figure out how to
55:21
do, but they can't lose the helium that's in there. They put it in there, it's sealed in, and it's not like they come around and plug in a thing every so often and add some more helium. Yeah, there was another question. So we actually wrote our own
55:43
erasure code. So it's like RAID, but it's I'll just say it's like RAID, but it's different. We did publish that and how we did that and we put it up on GitHub if you want to read it. But it is that same kind of a notion of sharding something across
56:03
X number of devices and having them be able to use, have so many of them to restore the entire thing. And so it is that notion of what RAID is. We used originally RAID 6 and so a lot of the storage pods still run RAID 6, but all of the vaults run
56:23
our own erasure coding. Anything else? How are we doing on? One more, last one. Fun question. If you're looking at S3
56:45
they charge a little bit more than we do. How much of that is profit, right? So I know they make a lot of money and Jeff Bezos is making more than our CEO. Now,
57:01
they have scaled to enormous areas, so that certainly adds cost to some level or another. They also subsidize some of their other businesses with the money they make and all of that. I think we do a really good job. They do some things better. They have a lot of compute capabilities and all of that, so I'm not
57:22
going to tell you we're the same service, but for what we do we try to make it as economical as possible and we'll always do that. Even though our CEO will be poor. Alright, thank you very much.