Faban: Developing benchmarks and workloads using Faban 1.0
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 97 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/45702 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 201073 / 97
1
2
4
5
6
8
14
15
16
17
23
29
38
41
42
44
46
47
48
50
53
54
62
63
64
65
66
71
74
75
77
78
79
80
82
84
85
94
00:00
BenchmarkMathematical analysisGraph (mathematics)Software frameworkOpen sourceProcess (computing)Data modelEuclidean vectorAutomationStatisticsSource codeOpen sourceInformation managementPhysical systemMultiplicationSpecial unitary groupUniform resource locatorConnectivity (graph theory)Mathematical analysisSoftware testingFile Transfer ProtocolCodeResultantMultitier architectureBenchmarkSoftware frameworkServer (computing)Service (economics)Endliche ModelltheorieProcess (computing)WorkloadElectronic program guideOpen setComputer animationXML
01:43
Mountain passMenu (computing)Graph (mathematics)StatisticsPhysical systemResponse time (technology)Physical systemStandard deviationStatisticsNumberBenchmarkMixed realityOperator (mathematics)Cartesian coordinate systemLatent heatInteractive televisionDependent and independent variablesServer (computing)ResultantGraph (mathematics)Key (cryptography)FrequencyMetric systemTerm (mathematics)Event horizonBefehlsprozessorElectronic visual displayWeb pageTraffic reportingTable (information)CASE <Informatik>Web 2.0Type theoryHome pageLoginVideoconferencingUtility softwareMaxima and minimaPrice indexAverageComputer animation
03:54
ArchitectureVirtual machineProxy serverPhysical systemInstance (computer science)CuboidSoftware testingEntire functionWeb 2.0Run time (program lifecycle phase)Connectivity (graph theory)MereologyStructural loadComputer architectureCASE <Informatik>Server (computing)DatabaseComputer animation
04:51
BenchmarkConfiguration spaceSystem programmingSoftware testingServer (computing)Instance (computer science)CodePhysical systemData analysisStatisticsConfiguration spaceTouchscreenInformationWeb 2.0Cartesian coordinate systemDatabasePhysical systemComputer fileReverse engineeringBenchmarkSampling (statistics)CodeMultilaterationSlide ruleComputer simulationStatisticsTable (information)Form (programming)BitoutputInformation managementPresentation of a groupSoftware testingServer (computing)Proxy serverMereologyNumberInstance (computer science)Bookmark (World Wide Web)Computer animation
06:33
Configuration spaceService (economics)StatisticsComputer filePhysical systemData storage deviceSet (mathematics)Wrapper (data mining)FrequencyMeasurementLatent heatConfiguration spaceWeb pageDatabaseService (economics)NumberInstance (computer science)Server (computing)CASE <Informatik>Software testingMobile appFront and back endsBenchmarkInformationWeb 2.0Run time (program lifecycle phase)Steady state (chemistry)Multiplication signProxy serverJava appletTouchscreenBootingCartesian coordinate systemCuboidBitStructural loadCycle (graph theory)Dean numberData loggerProcess (computing)Video gameComputer animation
09:41
Control flowProcess (computing)BlogImplementationService (economics)Cycle (graph theory)BenchmarkService (economics)Configuration spaceSteady state (chemistry)Response time (technology)Video gamePhase transitionFrequencyBootingThread (computing)Point (geometry)MeasurementPhysical systemSource codeProcess (computing)Dependent and independent variablesRight angleStructural loadComputer animation
10:39
Military operationSocial classJava appletWorkloadOperations researchWeb pageComputer-generated imageryFluid staticsStructural loadCodeCycle (graph theory)Home pageEvent horizonLoginRevision controlScale (map)Software frameworkException handlingWeb pageServer (computing)Medical imagingCodeOperator (mathematics)Physical systemStructural loadSocial classSoftware testingJava appletTouchscreenObject (grammar)BitEvent horizonSampling (statistics)Mixed realityGroup actionSequenceMultiplication signFinitismusNegative numberChainPotenz <Mathematik>Dependent and independent variables2 (number)Maxima and minimaSystem callCycle (graph theory)Graph (mathematics)Exception handlingInteractive televisionCASE <Informatik>InformationComputer fileForm (programming)State of matterGreatest elementSpacetimeArithmetic meanMetadataRow (database)Home pageWeightLevel (video gaming)Parameter (computer programming)Matrix (mathematics)Computer programmingMehrplatzsystemFehlerschrankeRandomizationPersonal digital assistantRight angleStandard deviationCache (computing)Computer simulation1 (number)Computer animationLecture/Conference
15:30
XML
Transcript: English(auto-generated)
00:09
Hi everyone. So yeah, so talking about developing benchmarks and workloads using Fabin 1.0. My name's Amanda Waite, I work for Sun in the UK.
00:25
Okay, so we're talking about Fabin. Anybody use Fabin here? Okay, alright, so that's what I assumed. I didn't expect many people would have used it, so we're going to kind of give you the beginner's 101 guide to it. So Fabin is a facility for developing and running multi-tier server benchmarks
00:43
across multiple systems. You can also use it for doing microbenchmarking as well, maybe a specific component such as an FTP server or a HTTP server. It's open source, it's developed by Sun, and it's available at that particular URL,
01:00
which I'll give you a reminder of later at the end of the slides. Fabin has two major components, the harness, the harness, which automates the whole process of a benchmark run, and also provides a container for your benchmark driver code. Again, we'll look at some examples of benchmark driver code in a minute. We have a driver framework, which provides an API for you to develop
01:25
benchmark drivers, also services and tools, and also provides a component model. And also, I mentioned there's two main aspects to it, but there's also analysis as well. So Fabin provides comprehensive analysis of results and data it collects
01:41
from the systems involved in the test. So the best way to introduce Fabin is to show you some of what it does and what it gives you. This is a typical Fabin benchmark run. You'll see at the top here, we have information such as the operations per second for this run, the number of operations that were run, and whether it passed or failed.
02:04
And beneath that, you have several tables, and two of the tables are displayed here. An important concept in Fabin is the operation. I'll mention operations quite frequently. An operation is basically a specific use case.
02:20
It acts as interaction with a system, for example, a user accessing a web server or web page. The two tables we have displayed here are the operation mix, which shows you the number of operations that were performed successfully, the number that failed, and the actual mix that we had of that particular type of operation.
02:43
Now, what we're seeing here, this particular benchmark run, was for a social networking application. So for a social networking application, we have operations such as home page, login, add event, or event detail. They're all of these operations, specific user interactions with the system.
03:02
Down here, we have response times. We have response times for all of the operations involved, average, maximum, 90th percentile response time, and standard deviation. Fabin also gives you a detailed results page with graphing of key performance indicators,
03:20
things like throughput and response times. There are a few others as well about what the drive has done and what work it's done to perform these particular metrics. And depending on what you've done in terms of deploying tools, across the systems in your benchmark run, you'll get specific stats for each system. You can drill down for each system, look at statistics such as CPU utilization,
03:43
or look at the NIC and see what kind of throughput is going through the NIC. That's the kind of stuff you get from Fabin in terms of the graphical display and the reporting. The Fabin architecture, we have two major components, two major runtime components within Fabin.
04:01
We have a master. The master coordinates all of the systems involved in the test. And the other aspect of it, the other part of it, is the agent. And we have several agents running on the other systems that are actually involved within the test. So in this case, at the top here, we see a bunch of driver systems.
04:21
They're actually JVMs. They don't need to be specific systems, so you may find these three running on the same machine. They're instances of a JVM. Each one has an agent, and these machines are driving load into the systems on the test down here. This gray box contains the entire system that's on the test. So we have web servers, proxy servers maybe at the front,
04:41
and we also have databases, memcached instances, those kind of things. So all of those systems will be running agents, and Fabin can control them all through the master. You have to give Fabin certain information to get everything working. You have to provide information about all of the systems involved in the test. So, for example, the web servers, the application servers, database servers,
05:05
memcached instances, reverse proxies, host port pairs for the most part. You also have to provide a benchmark driver, but Fabin gives you some samples you can use to develop that. And we'll look at some of the code that you would use to develop a benchmark driver in later slides.
05:23
And you also have to provide a configuration. So the configuration data is details about what the run will do, how long it will run for, the number of simulated users, et cetera. So Fabin gives you back some of the stuff we've already mentioned. It gives you a harness that runs all your driver code for you,
05:41
that manages all of the systems within the run, that gathers the data at the end of the run, makes sure the statistic tools are running on all of the systems, brings it all together at the end and presents you the data in graphical and tabular form. It also provides you a driver API, along with all of the plumbing to run your Fabin benchmark driver.
06:05
And we'll talk a little bit more about that in another slide. You also get configuration screens, where you provide all of the input data about the systems involved in the test, and we've got a couple of examples in a second. These are all also generated from a configuration file,
06:21
a very basic configuration file that you provide, and when you deploy the benchmark, the configuration screens are all generated for you. An example of that, this is one of four tabs in a configuration. So you see at the top here we have four tabs.
06:42
The Java tab here allows you to configure the Java runtime. The driver tab is where you provide information about the run, how many simulated users, what the steady state time will be, how many driver agents will run, how many systems will be generating load, that kind of thing.
07:00
And this screen is the data service tab. This is where you configure all of the backend systems, and in this case we have a database server, a file store, and some memcached instances, although we haven't provided any data about them. So that means we've probably not used any on this particular test. We also have the ability to say whether we want to reload
07:21
these particular resources at the beginning of each run. Also we have opportunity to provide fading with information about where the configuration files are, where the log files are, and how to stop and start the services. And also what tools we want to run for any particular given run.
07:46
On the web server configuration tab, in this case we're running the OLLO, or we're testing the OLLO, Apache OLLO social networking application, and we're using the Rails service and the Nginx service. So in this case we get two boxes, we get one for the app server,
08:03
and we get one for the proxy server. So here we're using Fin, this is the Rails service, so we're saying we want to use the Fin backends. We're going to run 32 of them. The starting port will be 8,000. And here we're saying we want to use Nginx. And what the port is. And then we provide all this other information again
08:20
about how to start and stop those particular processes, and where to get the log files, where to get the configuration data and such like. And also again we can provide information about what tools we want to run. So services and tools. A service is something that you want to run
08:40
throughout the duration of the benchmark run. It needs to be in place before the benchmark can actually start. And that could be something like a MySQL instance, initialised with a set of data that you've provided. You generally don't start testing with a raw database, you'll load the database up initially. It could also be the number of...
09:00
a service that provides like Fin instances like we've seen before on a previous configuration page. Or a service that initialises a file store with clean data for each run. And it also provides a pluggable infrastructure for you to develop your own. And it also provides a basic set of services and tools that you can use.
09:23
I forgot to mention what a tool is. A tool is something like a wrapper around VMstat that will run during the steady state period of a run to measure specific statistics about the system. A monitoring tool, something maybe like a memcached.stats tool, that kind of thing.
09:43
So the life cycle of a Fade and Benchmark run, as far as services and tools are concerned, we start off with a configuration phase and we end up with a post-processing phase. In between, we start off our run with a ramp-up period where we're starting all of the loader threads, all of the driver threads
10:01
to simulate the load. We're warming up caches and such like until we get to a point where we're in steady state. And this is the point where we want to actually do the measurements. We actually want to test or measure the performance, the response times and the throughputs during steady state. So what happens with a tool, a tool starts only in steady state which you don't really want to measure the system
10:21
while it's not in steady state. So you wait to this point here and then you start all of the tools. So all of the tools will run until the actual steady state period ends with a ramp down here. Services have to be in place right from the beginning so they start here and they end in the post-process phase.
10:41
So looking at the Fade and Driver, oh, I do apologise. We seem to have run out of screen. So a Fade and Driver is a Java class that's used to drive load to your systems on the test.
11:01
It's something you write. It uses Java annotations. It's basically a plain old Java object but annotated with Java annotations provided by the Fade API. It uses these annotations to define metadata and also define operations which we talked about earlier.
11:20
So an operation is a specific interaction with the system. It may well be that in this case you get a HTML page initially. You do the GET and then you load all the static information that you need to render that page, the JavaScript files, the CSS files and such like. You pull them back from the server.
11:41
Then you scrape the HTML to find out what images are on the page and then you can load them from the server or you can load them from local cache if you're simulating a cache. Another example would be for a form data, HCTP post request, the response code may be a redirect in which case the operation may act on a redirect
12:01
and redirect the user to another page. Not effectively viewing the page but the simulated user itself. And it also validates the success of the operation. So the operation fails, it throws an exception and Fade registers that as a failure. Down here we have an operation mix.
12:21
So you can't see this. I think it's quite a bit underneath there. But you'll see it on the sample code. But an operation mix is basically the transition from one operation to another. So you perform an operation or a user would perform an operation. Then you'll do something else on your system. So an operation mix defines how that happens. It could be a simple sequence
12:41
like do this, then do that, and then do that. A sequence is constantly repeated by every single user. Or it could be something more complicated and that's where we start talking about using finite Markov chains to simulate randomness within the system. But you can do all of that and you get that absolutely for nothing. It's a bit more complicated to program that
13:01
as you'll see on the next page. So this is a typical, not necessarily typical, but this is a Fade and Driver class annotated. At the bottom there, I don't know if you guys can see it, is the class itself, UiDriver. And then we have a whole bunch of metadata annotations that we provide.
13:21
The first two annotations are purely metadata. We won't need to go into those, but they're the ones you provide to the system. And then we provide a matrix mix. And we mentioned operation mixes before on the previous page. This could be something very simple. It could be just a very simple transition from one operation to another. Or you can use some real intelligence in it
13:41
and you can use a matrix mix. This takes two arguments. It takes an array of operations and it takes a mix, which is a matrix. And we have one row for every operation and one column for every operation. And this allows you to define the state transition from one operation to another.
14:02
And also, a weighting of what would be the preferred operation to run at the next stage. The next operation run, maybe we've got a heavy weight here that after we've run the homepage, we'll do an event detail. And also, using this, you can actually prevent certain state transitions from happening.
14:20
At the bottom, we have this thing, a negative exponential. And we're using this thing called a cycle time. A cycle time is a space between one operation and the start of one operation, and the start of the next. You can also define a think time. We're saying we want to run with a cycle mean of 5,000 milliseconds with a 2% deviation.
14:41
And we want it to be a negative exponential. So Fabian will actually vary the cycle time between operations to fit a negative exponential response graph. So some of them may be 25 seconds, some of them may be zero, but it will get the average right to be about five. And a couple of methods, annotated.
15:00
So we have doHomepage, which is a homepage operation. And we have doAddEvent, which is an add event operation. We can say the maximum 90th percentile time, automatic timing, so Fabian will start the timing when we enter the method. This one here is using manual timing. We enter this method when we call this event.
15:22
And if that's given you any kind of interest, please go here and find Fabian, because it's really cool.