Tweaking Ruby GC Parameters for Fun, Speed, and Profit
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 87 | |
| Number of Parts | 89 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/31572 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
RailsConf 201687 / 89
1
2
3
5
7
8
11
12
19
22
28
29
30
31
33
45
47
50
52
53
58
62
67
71
74
76
77
78
79
80
81
84
85
86
88
89
00:00
Right angle2 (number)Electric generatorGenderoutputSemantics (computer science)Product (business)MereologyMultiplication signObject (grammar)1 (number)ResultantInformationBeat (acoustics)SpeicherbereinigungAlgorithmForm (programming)ArmDifferent (Kate Ryan album)Sweep line algorithmLatin squareParameter (computer programming)BlogCodeCondition numberPoint (geometry)DialectDisk read-and-write headMedical imagingAuditory maskingLink (knot theory)BitPerspective (visual)MassLogicCartesian coordinate systemMemory managementGroup actionFamilyPhase transitionIntegrated development environmentPhysicalismError messagePresentation of a groupWindowBoss CorporationDecision tree learningCodeTunisTerm (mathematics)Computer clusterStapeldateiSoftware developerGoodness of fitRaster graphicsConfiguration spaceRuby on RailsSymbol tableComputer configurationMobile appDatabase transactionAverageDemosceneComputer animationMeeting/Interview
08:35
Object (grammar)AverageSystem callElectric generatorAlgorithmContent (media)NumberMultiplication signSoftware bugService (economics)Parameter (computer programming)Symbol tableAsynchronous Transfer ModeVirtual machineCartesian coordinate systemConfiguration spaceMobile appHost Identity ProtocolPhase transitionComplex (psychology)SpeicherbereinigungFreeware2 (number)Arithmetic meanMaxima and minimaResponse time (technology)Source codeDenial-of-service attackMemory managementSheaf (mathematics)Uniform resource locatorDivisorLimit (category theory)Set (mathematics)Variable (mathematics)Integrated development environmentRevision controlSoftware testingDescriptive statisticsDefault (computer science)POKEDifferent (Kate Ryan album)Decision tree learningComputer configurationMathematicsSemiconductor memoryPropagatorLaptopMereologySupremumStructural loadComputer fileAdditionElectronic mailing listArmQuicksortImplementationExecution unitFamilyParticle systemInformationGreatest common divisorReading (process)Lie groupSpring (hydrology)FreezingMusical ensembleInstance (computer science)Point (geometry)Vector spaceHypermediaPositional notationAffine spaceGradientStudent's t-testMatching (graph theory)
16:52
NumberCausalityInformationLaptopMultiplication signSoftware testingPressureRight angleIntegrated development environmentNumbering schemePower (physics)Dynamical systemMathematicsMereologyTask (computing)Cartesian coordinate systemLevel (video gaming)Directed graphSimilarity (geometry)Sound effectGame theoryShared memoryCASE <Informatik>Lattice (order)ArmSeries (mathematics)DecimalMusical ensemblePresentation of a groupPoint (geometry)MeasurementParameter (computer programming)BitSystem callAverageGoodness of fitComputer architectureResponse time (technology)Process (computing)Product (business)DatabaseHypermediaPredictabilityDisk read-and-write headOnline helpDifferent (Kate Ryan album)Focus (optics)Object (grammar)Database transactionRange (statistics)Dependent and independent variablesSpeicherbereinigungUniform resource locatorFront and back endsConfiguration spaceReplication (computing)TheoryBoss CorporationStructural loadAlgorithmMemory managementMobile appReading (process)Server (computing)DemosceneComputer animation
25:09
Speicherbereinigung1 (number)Expected valueLattice (order)Multiplication signCartesian coordinate systemCASE <Informatik>Absolute valueClosed setDatabase transactionPropagatorRight angleError messageProduct (business)AverageDifferent (Kate Ryan album)InformationCivil engineeringBitPresentation of a groupShared memoryWeb 2.0Response time (technology)TrailDrop (liquid)DistanceBoss CorporationMedical imagingQuicksortMetric systemFreezingYouTubeArmLimit (category theory)Suite (music)Maxima and minimaSheaf (mathematics)Software testingDemosceneRow (database)WordAdditionLogic gateNeuroinformatikIntegrated development environmentWebsiteAvatar (2009 film)Noise (electronics)Level (video gaming)MathematicsSymmetric matrixCondition numberCellular automatonMobile appDependent and independent variablesSampling (statistics)AlgorithmGame theoryFreewareReal numberParameter (computer programming)LaptopConfiguration spaceRevision controlSpeech synthesisFeedbackStructural loadGreatest elementComputer animation
33:26
Level (video gaming)NumberProduct (business)MathematicsIntegrated development environmentCartesian coordinate systemRange (statistics)Different (Kate Ryan album)Semiconductor memorySampling (statistics)Mobile appSet (mathematics)1 (number)Object (grammar)Revision controlSource codeSlide ruleWeb 2.0PhysicistCombinational logicPattern languageResponse time (technology)Task (computing)Configuration spaceSoftware testingComputer configurationMultiplication signParameter (computer programming)Default (computer science)SpeicherbereinigungMemory managementPoint (geometry)Dependent and independent variablesResultantBitDatabase transactionStructural loadPressureGodPresentation of a groupSoftware developerRadical (chemistry)Real numberReplication (computing)Price indexArmStress (mechanics)Power (physics)Surjective functionUtility softwareDemoscenePole (complex analysis)ChainGroup actionDialectInheritance (object-oriented programming)Online helpInformation securityCollisionQuicksortCivil engineeringSymbol tableForm (programming)PropagatorNoise (electronics)Forcing (mathematics)Water vaporVirtual machineSystem callDivisorPersonal digital assistantComputer animation
41:42
Computer animation
Transcript: English(auto-generated)
00:12
10 seconds of my timer here so I think I'm going to get started. So hello everybody. So the title of my talk is that when it can read in the end tweaking will be a garbage
00:22
collector parameters for firm speed and profit. My part is the firm part for my boss is really speed and profit. So that's how I got to convince him to give me time to work on that. So before I start, let me just tell a little bit of story here. So who is here, all right, all right, it's better?
00:48
Okay, I'm gonna try to stay still in here. So who is here for the very first time? All right, it's a pretty good audience and that's something that's really amazing to me about RailsConf.
01:04
Last year was my first RailsConf and I had a really great time. I thought it was awesome conference. And after it finished, I decided that this year I wanted to come here as a speaker, and as a matter of fact, I'm here. So if any of you guys that are here for the very first time,
01:22
if you like the conference, and after that, you think it's really awesome. You're not alone, a lot of people is gonna think like that. But if you decided to come next year as a speaker, I can assure you that's totally possible. And if you have any questions about that, if you wanna talk, if you really got interested to that, let me know.
01:41
I can share a lot of the road that took me here. And so with that out of the way, my name is Elio Cola. I've been doing software development for about 15 years. I spent about ten years doing C, C++, Solaris, Linux environment, before I switched to Ruby on Rails.
02:00
It's been about five, six years that I've been working for Ruby on Rails. And I go by that thing on the Internet, if you wanna find me. So all right, let's get started and let's talk about the Ruby garbage collector. So throughout the presentation, I'm gonna go through some of these terms. GC, garbage collector. IGMDC is the restricted generational garbage collector.
02:24
The RinkDC, it's the restricted incremental garbage collector. COW is copyright technique. AB is a batch plain mark tool. Just a couple of terms. My talk is gonna follow these topics. So while I'm here talking about garbage collector,
02:40
despite my little story at the beginning. Something a little bit about the history of how the Ruby garbage collector algorithm evolved through the Ruby releases. And some configuration parameters, how they evolved. And my approach into measuring and tuning this thing. These options are gonna be how I got to this talk,
03:02
how I created all these, and how I learned about all these. And at the end, I hope I have some time for some questions and answers. All right, so why I'm here and why garbage collector? So also upon a time, there was a Rails app. There was one, two, three, four, five Rails app. And they were all in production, living, live, and prosper.
03:24
But we didn't have a lot of inside story on how they were behaving. So we, proactively, in the company, decided to install one of these full fancy, full offense charts, monitoring tools. And then that's how all these got started. We installed these monitoring tool and
03:42
we start seeing, I got exposed to this information. And I was looking to those things and I was like, all right. On the left side, I have everything mostly blue. It only says GC runs. On the right side, I have these mostly yellow, a little bit of blue. And it shows minor and major.
04:01
So, hm, what is that? I don't know, but it sounds like something interesting to find out. And I also saw these on the left side. You can see that the average is 80 times. That's how many times the GC is running per hundred transaction and the right side is running 46. Like, hm, all right, the right side seems better.
04:21
So why my left side app is not behaving as my right side app? So I wanna find that out. And up until that time, I have enough questions that got me going. And got me interested in these. So I did a lot of research. I read a lot. I googled everything I could find about Ruby Garbage Collector.
04:41
And I had a really great time. I really, really have a great time learning all that. It was pretty awesome. I guess I was a little bit bored at work as well. But, and then I decided that I wanted to share that. Because I didn't, despite having found a lot of documentation, I didn't see a lot of people talking about the Ruby Garbage Collector.
05:01
And finally, my talk got accepted, right? So if that didn't happen, I wouldn't be here, so. So I'm gonna tell, before I dive into how I approach these things, I'm just gonna give you guys a little glance on how the algorithms evolved through the Ruby releases.
05:21
So let me ask a question before here. So how many of you guys here ever changed tune in Ruby Garbage Collector in production? All right, that is a handful, a dozen people. All right, that's interesting. One of the meetups that I did in DC, I think one guy in the room, but
05:40
it was a much smaller room anyway. So this part of the presentation, there's gonna be a lot of information. I don't expect you guys to follow everything, to get everything on your mind. I'm only gonna glance through these algorithms. I'm not gonna explain them in details. But if that's something that you got interested in, you get curious about that, just let me know.
06:01
There is a lot of documentation that I can point you out. You can Google around it as well and find that. But I have some reference that you guys can use and I can point that to you. So basically 1.8, the algorithm is simple marker sweep. We're going to use the lazy sweep, 2.0 and the bitmap masking.
06:22
The Ruby 2.1 is the gem DC. Ruby 2.2 come with the ink DC and symbol DC. So during my research, I came across a mom's good blog. And I didn't came across this blog at the beginning of my research. It was close to the end.
06:41
It was after I read a lot about the Ruby garbage collector, mostly about the gen DC and ink DC, the generation incremental garbage collector. That's where you find more documentation about it. And then towards the end of my research, I came across this blog and I am a visual person.
07:01
And when I saw these and I saw how he concisely and visually expressed the difference between these different algorithms, it all clicked in my head. And it also gave me some perspective on how prior to 2.1 and 2.0x, how those algorithms evolved. And so for the next couple slides, I'm just gonna show a couple of screenshots
07:20
from his blog on how he showed that in a visual way. It should be used to digest even though there is a lot of behind the scenes. So 1.8 is simple mark and sweep. So basically you have a mark phase and sweep phase. During that time, they just stop the road. That means that your code's not running because the garbage cart is running.
07:42
And it marks a bunch of objects, and then the ones that are not B. Being referenced, they got sweeped, and then your code resumes. 193 is a lazy sweep. The main difference in here is the mark phase is still the same, but the sweep, it's doing lazily. So as objects are required, they are released, and they are swapped for
08:04
you to use, and that's the 193 lazy sweep. 2.0 bitmap masking, this is just about memory management. It's improved memory management for our gems and applications that do fork and do those kind of things. And also the mark phase is rewritten to be non-recursive.
08:23
That's what you got in there, but the main logic about the mark and sweep phase remain the same. And if you plug a 2.0 application in one of these fancy, full of fancy charts, monitoring tool, that's what you see. You see GC runs. This is from a simple application that I have on my machine.
08:42
Ruby 2.1, here's where the juice starts. So generation garbage collector. The idea here that objects die young, you probably heard about that. If an object survives a GC run, it gets promoted to old generation objects. So next time it run a minor GC,
09:01
it doesn't go through all the objects, only the young objects. So you traverse a smaller list of objects. So we spend less time. And that's kind of the whole idea of the generation GC. Again, there is a lot more into that. I'm just gonna go into these to give you guys some basics.
09:23
The 2.2 has the ink GC and the symbol GC. Symbol GC, hey, just get collected now. So if you're a symbol lover, you don't run the risk of a DOS attack crashing your app anymore. So the symbols are gonna get collected.
09:40
And the incremental DC. So the incremental DC, it's all about shortening the pause times. As you can see, it's just a replacement of a long mark pause is replaced by a few small mark phases. And from the implementer, which I think is here in the room, this is not a silver bullet.
10:04
It's not a guarantee that's gonna increase your throughput and improve your response time. So it was all about replacing a long pause to a small pause. And if you plug a 2.1 and 2.2 application and
10:21
also 2.3 into one of these four fancy charts monitoring tool, that's what you see. You see major and minor. The major should run less often than the minor, and that's how it looks. So a couple of configuration parameters. The same way that the algorithm evolved and got more complex and
10:42
faster, also the configuration parameters, they evolved. They grow in number and complexity. In the 2.0, we have these three configuration parameters, malloc limit, hip mean, slots, and free mean. This is all about how many slots you allocate during your application start up and how many slots needs to be free.
11:04
The minimum number of slots needs to be free after the GC runs. So it's gonna release a couple of memory, but there is not that amount of free slots. It's just gonna allocate more for your application to use. In Ruby 2.0x, we have these three parameters. As you grew to 2.1 and above, we have now 11 configuration parameters.
11:25
There is some documentation explanation about all these parameters. I'm not gonna go into details here because you're probably gonna leave the room if I do that. So, but basically, those are the parameters. Hip in each slot, hip free slot, hip growth factor, hip growth max slot,
11:43
hip old generation object limit factor. This is a pretty cool one. I kinda like it. So now you have these other three sets of configuration parameters that are related to a young and old generation objects. Limit, limit malloc, limit max, malloc, limit growth factor, and
12:03
these things. I didn't poke around all of them. I kinda changed most of them and tested during when I was doing all these work. But at the end, I didn't end up changing all of them. All of them had default values, so if you don't change, there's gonna be a default experience. But sometimes that is not your best option.
12:25
If you look at the Ruby source code, the gc.c file, and I have the URL for the references in there, that's what you're gonna see. On the QNX and above, you have this long C commented section that lists all the configuration parameters, some documentation.
12:42
And if you can read that, the first one, the init slots is initial allocation slots. The one that I thought was cool was the gc hip old object limit factor, which is do a full GC when the number of old objects is more than r times n. Where r is this factor and
13:01
n is the number of old objects just after last full GC. First of all, this description here, when I read that, I was like holy shit, that's pretty cool. So as I said, I worked for almost a decade with C and C++. So going back to these and reading these stuff and compiling the Ruby source code and making that compiled version
13:23
of Ruby be used on my Rails app, it was pretty awesome to me. Before that, I never really went back to C and C++, so that kinda helped me to have some fun during this process. And at the end, if you're wondering, that's how you actually test that.
13:41
So if you wanna change some of these configuration parameters and test them in your application, your laptop, that's what you do. Those are all environment variables on your Linux machine. So if you export that variable within some value and then do Rails S, then when the Ruby VM starts and the garbage cart gets set up, it's gonna pick up those variables from the environment and
14:03
it's gonna create the memories accordingly to that. If you wanna change them, you stop Rails C, export these values for different value and restart it again. So for testing your laptop, that's pretty much what you do. So with that, I finished all this stuff that I wanted to talk at
14:22
a glance to you. If those algorithms, details about it, is something that you get curious and you're curious to learn more about it, I can point you guys to some documentation. While also I strongly recommend you guys check out Eric Weisten's talk, RubyConf last year. He gave a talk on garbage content as well, but
14:41
his approach was explaining more than two details how do all these algorithms work. So I was here, I saw his talk, and I watched his talk after I've been to all these, learning about all these stuffs, so watching him explain these algorithms was pretty cool. I really like it. I really felt like I was back in college and
15:01
watching my CS teacher explaining algorithms. It was pretty cool, I like it. And as a matter of fact, I don't like everything, but the things that I'm talking about, probably there is a reason for that. And so I think his talk was pretty cool, I really like it.
15:21
So now let's go back here to where everything started. So as I mentioned, I got exposed to these things that you see. The left side you have the GC runs, and the right side you have major and minor, and those average on top 80 versus 40. It kind of bugged me. I was like, hm, that sounds like there is something here for me to improve.
15:43
But I wasn't sure whether those numbers were normal. Maybe I'm in a normal scenario. During my research, I couldn't find any documentation or anything that gave me a clue that all right, if you're running an average from 0 to 15, you are awesome. If you're from 15 to 25, you're great.
16:01
25 to 75 is okay. And above 75, you're really bad. I couldn't find anything or get to any knowledge about that. So I wasn't sure whether my 46 was great or was really bad. Maybe tomorrow I'm gonna get a call overnight saying that my API is down. The service is down and I'm gonna have to wake up in the middle of the night
16:21
to fix this application and try to find out why the machine is dying. I wasn't sure. And that was one of the reasons that why we set up, we installed this monitoring tool on our application. Because we wanna get a more Intel and more practical measure and see how healthy our applications were. And so I went on research mode.
16:42
That's what got into me all this. I read, I studied all the source code. I compiled the Ruby source code. I did all of that. And as I mentioned, I haven't really got a great time. And after a lot of research, and I realized, all right, that Ruby 2.0x,
17:03
I'm not gonna worry about that. That's gonna get into upgrade path. It's gonna go to 2.1.2.2. In the end, the 1.2.2, all right, let me find out what I'm gonna do. But that application, 2.1.x, as I mentioned, I didn't know whether 46 was normal. Maybe 46 is super great. I'm in a really good spot. I don't need to worry about this, but I wasn't sure.
17:22
I needed to find out. So what did I do? All right, and then I thought to myself, all right, let me run all these on my laptop, do some tests, and do some load tests, some basic load tests. I'm not gonna go too crazy, not spend too much time. And maybe I'm gonna find out something. Maybe that's gonna give me some clue to whether 40, it's good or
17:42
it's bad, or maybe I'm gonna realize that I'm just crazy and go do something else. So I did that. I did a lot of tests. And at the end, that's what happened. I was still on my laptop, despite everything that I do. The GC was running 3.99 on average. I was like, hm, all right.
18:00
So it's not like what I'm seeing in my production apps. My production apps are running 40 times out of, on average, 40 times out of 100 transactions. And then what did I do? A couple more days go by, testing, managing, analyzing data. And during that process, what I realized is all right.
18:20
So the test that I'm doing on my laptop, it's not the same as my AWS server. It's not the same Azure architecture of my production environment. Maybe the test that I'm doing on my staging environment, which is shared between a bunch of applications, has the similar architecture. So maybe those tests are gonna be better than what I do on my laptop.
18:43
I don't have the same amount of data. In production, I have a freaking amount of data, huge amount of data. And I didn't have that neither on my laptop nor on my staging environment. And my AB calls, maybe they are not reflecting exactly how my endpoints have been used for my users.
19:01
And so then I kind of got to the point, right. I'm not gonna try too much to get to simulate my production environment, how much pressure I'm putting GCU in production in my testing environment. Because I may never get there. And so what did I do? I slapped on it, said, all right, I'm doing too many tests,
19:23
I have too much stuff on my head, and I needed to go do something else. And my boss was bugging me about that other feature that I was working on. And then during those meetings, after those meetings, and a couple of hours hang out in the meeting room. And I tried to talk to some of my teammates about the thing that I was
19:42
trying to do and try to get a sense of, all right, maybe they are gonna say, dude, you're crazy, forget about this. Or maybe they're gonna say, yeah, it makes sense. All right, what you're saying, your theory kind of makes sense. So have you thought about this? Have you thought about that? Have you seen this or that? So I tried to talk to them and explain to them what I was doing, because maybe that's gonna help clear my mind and
20:01
give me an idea on what I'm gonna do next. And what really helped me after that was really to plan what I was gonna do. Because up to this point, I have a lot of information about how these algorithms work. The parameters that I can change, some of the parameters that I change and work and doesn't work, and it makes a difference,
20:22
it doesn't make any difference. So I've done a lot of tests and I had a lot of stuff. And planning what I was gonna do next and focus on that, it kind of helped me a lot to set myself on a path where I was able to find something that I thought was cool. So and then after that, either this or that's gonna happen.
20:41
Either you're gonna see something that you weren't seeing before. And it was there, right in my face. And I wasn't seeing that. So on my dev environment, the same fancy monitoring tool was showing that my object allocation per transaction is on a range of 13,000 objects. While in my production it was 94,000.
21:02
It's like, all right, that might be why the GC is not running as crazy as it is in production because my backend doesn't have a lot in my testing environment, I don't have the same amount of data. And so I'm manipulating less objects and I'm spitting out the JSON response much smaller than what it's doing.
21:20
Maybe my AB calls is not actually trying to get as much data as my real users. And then when I did that, what happened? I got to something that was very similar to what I was seeing on my production environment. And that was, I was still on my testing environment. So, and then I got to a point where, all right, I have an environment in here
21:42
that simulates the exact same GC pressures in the Ruby garbage collector. That now I can control, I can test, I can replicate. And if I make any change, I can find out whether that's gonna improve or not. If that's gonna release the pressures on the garbage collector or not.
22:01
So I was pretty happy about that. If that doesn't happen, if you can't find a scenario in your testing environment that kind of replicate in some level of similarity, what you have in your production environment, then you establish that baseline and then you work on that. And then you see that if you can make change on that, it's gonna improve or not.
22:22
But if you can replicate kind of, if you don't have a similar environment, what you see in your production environment, then you kind of have a stronger case to make to whether the change that I'm gonna do in that is gonna affect, is gonna have the same effect in your production environment. But anyway, so I did get into that on my, actually,
22:44
both on my laptop and on the staging environment when I added more data into my database and when I changed a little bit of how my A, B calls were. And after I got to that, then it's about change, measure, and analyze. Change, measure, analyze. Do that over and over again.
23:01
Don't do that when you're hungry, because it doesn't work. Do that early in the morning only. And analyze what you're reading. Try to read your scene. Remember, I spent a lot of days looking through those graphics, and I didn't see for a long time, I didn't see that my dev environment, I was allocating 13,000 objects in my production environment, it was 94.
23:21
So it was there, black and white, maybe kind of grayish and white. But I didn't see that. You have to analyze, especially if you generate a lot of data, you have to take your time to let your brain to digest all that information. And one thing that really helped me is some of the numbers that I was getting during my tasks, I literally wrote down in a post-it near my laptop.
23:43
And I kept reading, kept looking into that from time to time. These use case, these test scenario, I got these numbers. These other tests, I got these. And I literally wrote down there. And that kind of helped me to try to extract some knowledge from there and get some clue out of that. So, and also the other thing is, that's something that I always do,
24:02
is if you're testing these things, and trying to extract knowledge from the test that you're doing, try to do one change at a time. Because then you don't have to worry about, which change I did affect that? So if you do one at a time, it's kind of a little painful, tedious, but it's a good thing.
24:24
Again, this is not an advice to you. This is just something that worked for me in this particular environment, this particular scenario. You probably have all the different ways to test and to approach some of these things. But this is how I felt comfortable and how I was able to validate myself and
24:41
the things I was doing while I was doing and learning about this. So, again, either this or that will help, that's what I mentioned. And if you find something, in my case I did find a couple of configuration parameters that improved my response time, and I was really happy about that.
25:00
If that really happens to you in something that you are doing, investigating, document it nice, share with your team, make a small presentation inside your company about that. Make a small talk out of it, and maybe next year you're here, RailsConf, speaking about it. And if you do that, especially in the company that I was working at a time
25:20
after all that work that I've done, it was kind of pretty easy for me to deploy this stuff into production. Or if you don't find anything, then you still document something. If you went through a lot of time, if you tried a lot of different things and you still didn't work out, you still couldn't find anything, document the same way. Because maybe there's a lot of people in the world trying to do the same thing
25:42
before you or alongside you. And that if you share that information, maybe you're gonna save a lot of people's time. Or maybe you're gonna get some feedback from the community on what else you should do to solve the problem you're trying to solve. And so that's how I went by all this stuff.
26:02
And now from the next slides, they're all gonna be a couple of charts that is very easy. And if you guys don't wanna do some of these things, you can piggyback on some of these images and you can send that to your boss. I'm pretty sure your boss is gonna like these charts.
26:21
And maybe that helps you to convince them to let you poke around some of this stuff. At the end, these were the parameters that I changed. I'm sorry. At the end, these were the parameters that I changed that I made all the way to production. In each slot, free slot, malloc limit, and limit max.
26:41
I'm not sure if all of them actually one or two of these combined was the one that resulted into the improvements. But these are the ones that after long, a lot of testing, those are the ones that I decided to make into production. So now I'm just gonna show you guys a little bit of some comparison.
27:04
And some charts, some nice charts. At least I think they're nice. So I'm gonna compare Ruby 2.0x, 2.0.0 with 2.1.7. B4 and Intune, D4 experience. Imagine that you have an app with Ruby 2.0.0 and
27:23
you migrate that Ruby version to 2.1.7. That's what you should expect to see in your application. Then I'm gonna compare the same app with Ruby 2.1.7 before and after turning these configuration parameters. And then it's gonna be the time of 2.2.3 before and after.
27:43
And 2.1.7 and 2.2.3 after the tuning, just to find out how they compare and how those algorithms play a role into spending less time doing garbage collecting. So as you can see here, on the left side I can see, on the left side is Ruby 2.0.x, on the right side here it's Ruby 2.1.7.
28:02
And that's how everything started. I had two apps with two versions of Ruby. And in these cases, the same application when I run and I do some load testing with Ruby 2.0.0, and the same application after upgrading Ruby version to 2.1.7. And out of the gate, D4 experience,
28:22
the time the average of GC runs drop from 80 to 40. That's what you get just by upgrading 2.0 to 2.1x. And so here now is a different metric. Again, on top is 2.0.0, on the bottom is 2.1.7.
28:41
And this metric is the time spent in garbage collector for percentage of your wall clock time. I'm pretty sure somewhere there's a really nice explanation of wall clock time really means, but that's a metric that they show up. If that's something that you're curious,
29:01
just I have some more documentation about that as well. But basically, by changing the Ruby, it drops from 18% average to 3% average. Just remember, this is an average of a metric based on a percentage. So I have to keep our expectations regarding to this one.
29:22
But this is a more easier metric, web response time. This one is kind of very easy to digest. If you can see on top, the dark brown one, it's the time doing garbage collector. And you can see that on top, you pretty much spend close to 40 milliseconds per transaction doing garbage collector.
29:43
That's 40 milliseconds of your web response time. And if you change from your application from 2.0.0 to 2.1x, that drops to well below 20 milliseconds. So you're getting there close to 25, 30 milliseconds. And in this case, you can see that the web response time goes from 156
30:06
to 133 milliseconds. So you're totally improving your web response time. And so two and seven, before and after. Here, you can also see on the left side is before,
30:21
on the right side is after. And the average of runs goes down from 40 to 24. These data is from my sample application that I run to get these charts to put in this presentation. What I really got into my real application when I did all these, we went down from 40 something to 12, 13, 14.
30:41
Was some improvements a little bit better than what you're seeing here. And again, the same time spent on garbage collector. On top, two and seven before, at the bottom, two and seven after. And you see that the time spent on garbage collector drops from 3.28 to 2.38.
31:04
It's a small drop, it's a percentage metric, but it's a drop of close to 30%. And in this case, the web response time, it drops from 133 to 129.
31:20
That's real game application. Ruby 2.2.3 before and after. Then it's very similar to what we see on the 2.1.7. Remember that when I was showing and explaining the algorithm for the Ruby 2.2.x, that's where the incremental garbage collector got introduced.
31:45
But that wasn't about improving throughput or web response time. It was all about shortening pause times. And it wasn't guaranteed that the incremental DC would improve your throughput and web response time. And then we're gonna see that when I compare the 2.1.7 before,
32:04
2.1.7 after, and 2.2.3 after. And so 2.2.3 before and after, very similar to 2.1.7 before and after. The time spent in GC also drops from 2.38 to 1.4, a smaller drop in this case.
32:23
Web response time during these tests, it dropped from 1.37 to 1.30 milliseconds. And this is 2.1.7 before and after. Here you see that the 2.2.x is actually still a little bit worse than the 2.0.x.
32:42
That's probably just a little bit of noise in my test. You see that on the right side, the chart kind of goes up and down. So probably I was doing something else with my laptop when I was trying to test these scenarios. And what I really got into the real app in my test environment, is that when I tested 2.1.7 after comparing to 2.2.3 after,
33:05
I consistently got an improvement of about five, four, five, seven milliseconds of web response time. But again, the 2.2.x algorithm in incremental GC is not supposed,
33:23
it's not guaranteed that it's gonna give you a better throughput and response time, not always, it's not guaranteed. But I kind of felt like, because my results on those were consistently a little bit better, I mean five, six milliseconds, maybe it's just noise. But when it's consistently getting the same results, test after test,
33:41
run after run after run, I kind of felt that my application was in a sweet spot where the change for the incremental garbage collector did help me. Did help the application, right? And so this is 2.1x, so in this scenario I'm just comparing 2.1x before and after. And at 2.1x, 2.1.7 after, and 2.2.3 after, my God, so many numbers.
34:08
So it's dropped from 2.38 to comparing those two. One is 2.38%, and the other one is 1.4. Here the web response time, it barely changed.
34:20
This case, pretty much noise, most likely. And with that, I kind of conclude what I wanted to talk to you guys here today. If you have any questions, I'm more than happy to answer them. Thank you.
34:45
So the question was, if this comes with more memory usage, yes. So you're just telling the garbage collector to allocate more memory. In my case, that was happening. I increased the heap, and then I have more memory for my transactions to allocate 9,000 something objects.
35:00
And so the garbage collector runs less often, because there is more memory available for him to run. Yeah, so the question is, trading memory for time? Yeah, exactly. So since I have more memory and I find it in the machines, that's what I was doing. I was giving the Ruby VM more memory, so it has more room to operate.
35:20
And so you need to run the garbage collector less often. What tool? So the question is, what tool I use to measure the garbage collector? I use New Relic, and so when we install New Relic in our application, we start seeing all these things. And so I didn't use, actually, I didn't use New Relic to test. New Relic is how I was getting these results.
35:43
But what I used to do my load test, I just used a partial benchmark tool. I was just putting a small little load in my application to get the garbage collector to run with the same pressure that I was getting my production. So the question was if, so these charts that I have are indeed for
36:01
my simple application, it's not my real application. But what I get on my real application. So one of the applications in the staging environment, I got about 10% web response time increase. And in production, those numbers are a little smaller.
36:22
Production have way more caching than a staging environment. This was one of the APIs. In some other APIs, there was one that I couldn't see any change in production. I saw some change in the staging environment. 5, 6 to 10% improvement in the staging environment and production was the same.
36:41
And after the end of all these, pretty much all APIs are using Ruby 2.2X with these configuration parameters. There was actually one of the APIs that I used pretty much the same parameters for more than one application because they have the same range of object allocation, it was around 9,000.
37:02
But there was another application that I had to change because it has a memory footprint a little bit different from the others. So we used a different set of parameters. Okay, so the question was the before and after values. The after values are those the ones that I put in here. The before is the D4 experience.
37:22
Oh, yeah, okay, so I did it. Yeah, sorry, I didn't actually put the numbers in here. I can send you some of those. Yeah, yeah, I can send you some of the samples that I used for these ones. And the before was what is in the D4 version of Ruby. In the source code, in some of these references here,
37:42
you can find the D4, and I can point that to you. Maybe you can make a slide available and- Yeah, I can do that. Yeah, I can do that. I can definitely do that, put the D4 values in the ones I used for these. Yep, that's a pretty good question. So the question was, how did I change these parameters if I did it
38:03
systematically or if I changed to anything? At the beginning, I was, is that okay? So at the beginning, I was learning and testing and change all of them. After reading some of these parameters, some of them, they go along together. So if you're changing one, you have to change the other.
38:23
But because in the 2.1x, you have 11 parameters, and not all of them are very easy for you to digest and change, and to monitor the change, I didn't really use. I kinda got, after doing a lot of tests, I kinda get an idea on the ones I wanna change. But the interesting point about the question is, there is actually,
38:42
after I went through all these, I found some documentation about not everywhere is affordable for you to test, right? So if you're a physicist, working with a hydron, super collider, you can't test a lot. You have to be very careful with the tests that you're doing,
39:00
which patterns you're gonna use. And I was actually really curious on trying to investigate some of those algorithms and see if, from all those parameters, if there is a better, a sweeter combination that's gonna fit better for application. Never really get time to do that. So the answer is, at the end, I really used some of the parameters that I felt
39:22
was, it helped doing my tasks. See, I have a 90,000 something object allocation. So if I give memory to the Ruby machine, to allocate the memory, if I give enough memory to the Ruby VM to allocate those objects without having to run the GC all the time,
39:43
then at least for my application, it's gonna take care of. The GC is not gonna be necessary for me to run that often. And when I got to that point, I was satisfied with the value that I was getting for those parameters. I think you're absolutely correct. I got to, I presented this talk in a conference in Europe,
40:05
and I met a guy in there, was a physicist and a developer. My gosh. He, my gosh. So, and he actually told me about the Gushi tables, you know, orthogonal, orthogonal something.
40:24
I don't remember what I thought in my head, but he was going to that. He's the one that will mention about these examples where not in all industries you can test, it's affordable to test as many options. You have to be, you have to start very well,
40:41
which parameters are gonna change, and then you run your test. Because running the test is really costly. For me here, running these tests, changing these parameters is not costly. I can't afford doing that. But I'm pretty sure there is a better combination of parameters that would make it better. So the question was if I should suggest to change the default value for these parameters, and the answer is no, and that we shouldn't do that.
41:02
Because like you were saying, these parameters are based on what is your memory allocation. So if your application allocates 10,000 objects per transaction, the default value is all good for you. Maybe you're never gonna be in a point where you need these parameters to be tuned. So like they said in one of the other presentations,
41:26
maybe 99% of the Rails apps in production does not need to change these. But maybe you're in the 1%, and I actually worry. One of our apps were a couple. All right guys, thank you so much.