The Art of Performance Evaluation

Video in TIB AV-Portal: The Art of Performance Evaluation

Formal Metadata

The Art of Performance Evaluation
Title of Series
Number of Parts
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date
Production Place
Ottawa, Canada

Content Metadata

Subject Area
"Contrary to common belief, performance evaluation is an art." (Raj Jain, 1991) Successful performance evaluation may not be achieved with merely executing common benchmarking tools. This talk presents fundamental principles of performance evaluation and how you can put them into practice. Do you understand what exactly "pgbench" does? Is it appropriate workload for your performance evaluation goal? Common benchmarking tools like "pgbench" are handy for just comparing system A and system B, but if you intend to deeply understand the performance of your system, answers to these questions are critical. In order to conduct a meaningful performance evaluation, the methodology should be elegantly designed to meet the goal of the evaluation: choose metrics for the goal, and choose observation techniques for the metrics. Each step requires careful consideration and deep knowledge about the target system. It cannot be done mechanically. This is why performance evaluation is an art. This talk presents principles of designing performance evaluations and shows how you can put them into practice by introducing the speaker's experiences of performance evaluations with PostgreSQL.
Performance appraisal Performance appraisal Computer animation Universe (mathematics) Diagram Physical system
Moore's law Electronic data processing Computer animation Information Virtual machine Information Pole (complex analysis) Power (physics)
Metropolitan area network Key (cryptography) Information Transformation (genetics) File format Information systems Real number Mobile Web Virtual machine Mass Transformation (genetics) Power (physics) Moore's law Process (computing) Internetworking Computer animation Internetworking Set (mathematics) output Right angle Internet der Dinge Physical system Physical system
Metropolitan area network Mobile Web Database Transformation (genetics) Perspective (visual) Power (physics) Arithmetic mean Internetworking Computer animation Computer hardware Physical system Fundamental theorem of algebra Physical system
Point (geometry) Information Observational study Transformation (genetics) Connectivity (graph theory) Coprocessor Field (computer science) Mathematics Software Semiconductor memory Logic Computer hardware Computer hardware System programming Hard disk drive Nichtlineares Gleichungssystem Physical system
Point (geometry) Metropolitan area network Range (statistics) Fitness function Insertion loss Thermal expansion Performance appraisal Performance appraisal Computer animation Blog System programming Procedural programming Physical system
Computer network Mathematical analysis Data storage device Mereology Performance appraisal Arithmetic mean Performance appraisal Computer animation Read-only memory Computer hardware Kernel (computing) Befehlsprozessor Core dump Code refactoring Integrated development environment Right angle Freeware Simulation Middleware
Boss Corporation Forcing (mathematics) Data storage device Electronic mailing list Maxima and minima Computer network Data storage device Electronic mailing list Mereology Performance appraisal Goodness of fit Computer animation Read-only memory Kernel (computing) Befehlsprozessor Computer hardware Core dump System programming Software testing Simulation Middleware
Point (geometry) Magneto-optical drive Measurement System call Performance appraisal Data model Estimator Goodness of fit Computer animation Computer configuration Endliche Modelltheorie Metric system Physical system
Metropolitan area network Electronic program guide Computer Mathematical analysis Performance appraisal Performance appraisal Computer animation Core dump System programming Selectivity (electronic) Gamma function Physical system Physical system
Socket-Schnittstelle Forcing (mathematics) Mathematical analysis Port scanner Measurement Theory Cache (computing) Computer animation Read-only memory Semiconductor memory Speech synthesis Right angle Cycle (graph theory) Form (programming) Computer architecture Physical system
Socket-Schnittstelle Game controller Zoom lens Intel Dependent and independent variables Graph (mathematics) Range (statistics) Virtual machine Cache (computing) Process (computing) Cache (computing) Computer animation Read-only memory Personal digital assistant MiniDisc Physical system
Game controller Metropolitan area network Operations research Game controller Functional (mathematics) Observational study Open source Information Dependent and independent variables Flash memory Data storage device Process capability index Database Bit rate Cartesian coordinate system Software maintenance Coprocessor Power (physics) Frequency Computer animation Personal digital assistant Operator (mathematics) Quicksort Physical system
Area Metropolitan area network Graph (mathematics) Divisor Multiplication sign Mathematical singularity Physical law Expression Mereology Sphere Power (physics) Field (computer science) Power (physics) Frequency Performance appraisal Computer animation Touch typing Data structure Quicksort Simulation Physical system Fundamental theorem of algebra
Pulse (signal processing) Mereology Computer programming Performance appraisal Mathematics Performance appraisal Process (computing) Computer animation Authorization Procedural programming Simulation Physical system Surjective function
Data model Mechanism design Computer animation Key (cryptography) File format Multiplication sign Core dump Approximation Variable (mathematics) Physical system Form (programming)
Asynchronous Transfer Mode Email NP-hard Logical constant Bit Disk read-and-write head Statistics Measurement Variable (mathematics) Rotation Cylinder (geometry) Population density Computer animation Cylinder (geometry) Robotics Radius Hard disk drive MiniDisc Skalarproduktraum Family Form (programming)
Metropolitan area network Asynchronous Transfer Mode Graph (mathematics) Cartesian coordinate system Distance Measurement Variable (mathematics) Rotation Computer animation Personal digital assistant Endliche Modelltheorie Musical ensemble Form (programming)
Asynchronous Transfer Mode Game controller Presentation of a group Service (economics) Dependent and independent variables Execution unit Bit rate Data storage device Average Data model Response time (technology) Average Befehlsprozessor Endliche Modelltheorie Information security Physical system Condition number Form (programming) Game controller Metropolitan area network Service (economics) Distribution (mathematics) Server (computing) Expression Data storage device Computer Infinity Multilateration Befehlsprozessor Process (computing) Computer animation Query language Auditory masking MiniDisc Physical system
Asynchronous Transfer Mode Server (computing) Context awareness Implementation Maxima and minima Food energy Variable (mathematics) Twitter Data model Frequency Response time (technology) Goodness of fit Average Befehlsprozessor Electronic visual display Cuboid Endliche Modelltheorie Physical system Rotation Metropolitan area network Dependent and independent variables Multiplication Graph (mathematics) Lemma (mathematics) Expression Basis <Mathematik> Cartesian coordinate system Measurement Inclusion map Computer animation Personal digital assistant Order (biology) System programming Triangle Video game Right angle Resultant Pole (complex analysis)
Performance appraisal Data model Computer animation Different (Kate Ryan album) Endliche Modelltheorie Measurement Resultant
Electronic program guide Electronic program guide Data storage device Client (computing) Database transaction Data storage device Client (computing) Density of states Measurement Radical (chemistry) Benchmark Computer animation Integrated development environment Personal digital assistant Matrix (mathematics) Endliche Modelltheorie Integrated development environment Thermal conductivity Physical system Resultant Physical system
Game controller Dataflow Concurrency (computer science) Euler angles Server (computing) Data storage device Bit Data storage device Measurement Computer animation Integrated development environment Personal digital assistant Semiconductor memory Ideal (ethics) Integrated development environment Endliche Modelltheorie Block (periodic table) Physical system Physical system
Game controller State of matter Data storage device Parameter (computer programming) Number Data model Goodness of fit Operating system Ideal (ethics) Endliche Modelltheorie Social class Physical system Multiplication Server (computing) Data storage device Parameter (computer programming) Maxima and minima Port scanner Variable (mathematics) Measurement Performance appraisal Computer animation Personal digital assistant MiniDisc Right angle Metric system Physical system
Data model Asynchronous Transfer Mode Inclusion map Simulation Matching (graph theory) Computer animation Data storage device Endliche Modelltheorie Simulation Measurement Physical system Physical system
Asynchronous Transfer Mode Multiplication sign Execution unit Parsing Mathematical analysis Wave packet Hypothesis Performance appraisal Benchmark Term (mathematics) Single-precision floating-point format Software Ideal (ethics) Endliche Modelltheorie Physical system Metropolitan area network Simulation Turing test Graph (mathematics) Real number Mathematical analysis Approximation Performance appraisal Computer animation Computer hardware System programming Wireless LAN Simulation
Sensitivity analysis Functional (mathematics) Simulation Proper map Distribution (mathematics) Process capability index Bit Mathematical analysis Ripping Port scanner Measurement Hypothesis Wave packet Personal digital assistant Ideal (ethics) Endliche Modelltheorie Right angle Endliche Modelltheorie Wireless LAN Simulation Resultant Physical system
Point (geometry) Histogram Orientation (vector space) Direction (geometry) Insertion loss Mereology Computer programming Theory Formal language Web 2.0 Derivation (linguistics) Performance appraisal Authorization Physical system Covering space Simulation Constraint (mathematics) Theory of relativity Inheritance (object-oriented programming) Channel capacity Bit Line (geometry) Measurement Performance appraisal Radius Process (computing) Computer animation Rootkit Order (biology) System programming Video game Summierbarkeit Right angle Procedural programming Quicksort Simulation Resultant
so today I'm going to talk about the actual performance evaluation and finally University researcher at the University of Tokyo and I'm I'm doing there is isn't stories systems research so this OK is going to be out of its conception and opinionated took so I hope you could enjoy this diagram that I'm happy to I'm happy Feichtinger euro opinion on this topic so what was it
started with this Christian what is the goal of information technology what is the information technology my answer
is the information technology is transforming the computing power into the business or social values on you can just say users value for this and penetrating machine the 1st automatic data processing machines in the world and this is built for the national population census in the United States for that population only the very fundamental information for the nation and to actually this tabulating machine is body machinery in Pollock into a viable information
and performance is the key criterion of art this transformation if we think the so uh computing power is not necessarily machine I'm showing you what is the kind of the computing power so the national population census was done by human historically and tabulating machine addressed going into the performance of the the processing thing so they volutional information information technology is actually the history of this in the input in the lumen of this transformation right and
and to the that acknowledge friend it's still basically about formation of computing power into user value because data is a very very good example of that that attempt to extracting valuable information for the massive amounts of data or Internet Internet of Things or cyber-physical system this in an attempt to capture the real world with many sensors and doing something interesting key so but to date the data is the social values and the performance will be a thing is becoming more and more important so that from
technical perspective there system is the very important fundamentals of fundamentals of our transform meaning uh processing power into user value data with engineers very important today because they know how this works so very deep knowledge of database system is directly connected to the performance of the and
performance study of the nervous system is really into and maybe you have to know how hard disk works are as as the works of processor works how memory works home network works and of course you have to know how traditional works and even the whole business works OK so it's really uh and to inform prefix of hardware to logics of frication and that every component technology is changing and growing very laboratory in this field of IT and sometimes Architecture of Computing Systems is also drastically changes so learning every complaint technology is necessary but not sufficient so important point is that
all 3 important point is that equation as a way of learning how this transformation works how bitterly system a can get to work together and achieve performance that is
performance evaluation the performance evaluation is the key losses to understand performance of systems and the important point is the performance evaluation must be uh users value all the entities so every performance evaluation starts with fitting a goal the goal of which contributes to the use value and then there are evaluation procedure is designed and designed toward such goal
when talking about performance evaluation maybe many people think performance evaluation all let's benchmarking all I can do it with a expansion or RPG range for the discrete yeah that's all are you may you remind some broke costs and our fresh in you know the creator faster than during old-fashioned graphics the could be the base and exact that frankly speaking I didn't not performance evaluation yet
benchmarking is an important skills and is actually in the part of performance evaluation but this is just a means of performance evaluation and there
are many individuals skills with his uh his desire for performance evaluation by bats they not the core part of performance evaluation right for performance evaluation is goal will lead to full 6 so the fuel cell free of
orchestrating individual skills is the core part all of performance evaluation this take example
sorry for example if your bosses who wins faster storage for apples arrested at by so maybe in your goal might be to find the best price performance and this is the and rumbles about village is list all is is is available on the market then by all users this available on the market and try all races these with the bench 10 you don't find this is is the the good of this brute force that makes sense if you have if you have much much money and much science so but if you have a feel so feel performance evaluation again
it's not the 1st characterize the was grown and model performance with some performance metrics and is to meet the performance so if you have more than 1 you can estimate performance mistakes on data sheets and played
invited you model really is measurement of Ontario is is this so uh in this point you can you actually you measure the performance with the option is is and so that's what additional call from the model and you haven't now good precision model and now that the candidate is at the end of March And finally called from the detailed performance of candidate is sixties and you get the final without so good point all of these approaches you do not have to try all these on the market and a quite a quite animal it can be used in the future evaluation telescope and another
important aspect of performance evaluation is that uh go could be changed because performance evaluation is the closest of understanding performance food it's for might depend insight into the target system and sometimes it identifies more important goal of and go we really changed so keep aligned his kids considering wanted that important goal for you use of the value and then and then when the core is changed the orchestrate their individual skills for toward the goal of this is the philosophy of of of evaluation and it enables it to you to a meaningful performance evaluation of this philosophy it can be said
that the the out of 5 was evaluation this is very famous book about performance evaluation and it's it's uh performance evaluation is an act so and cannot produce mechanically and each requires intimate knowledge of target system and can constellation of of the selection of methods and tools so people how can we
developed are all philosophy of this is not so easy and analysis speaking and a cornerstone of philosophy is experience of lot but the important thing is silly behind experiences so silly forms the foundation of our experiences and experiences helps deepening the understanding of the theory for the chief trying this cycle of the a way to develop the performance of the developed world your own philosophy and thought that he
didn't solicitor I experienced useful for want of orientation so that it may introduce some examples of this the measurements latency of profit cache memory system in in way architecture each false if have a local memory and the forces that are connected with the interconnect right to be have full access to remote merely takes extort cycles is exhaustive use cycles all our
modern process that has a material cash FIL smaller caches faster like this so far it is really difficult to measure this great about it is possible for the maybe this is interesting graph for you all are another example
is the case with traditional discovery this discovery is meant for the range growth Enterprise Storage with 160 hardest rise and fall when it starts a my caused by exposure are indeed many regiments on this machine of war example sponsors response regret
and more interesting example is the performance of PCI Express crushed its full basidiospores for storage of I latency is very very low relative to to the existing storage devices and soul in this case latency of processor interconnect is very believable other portion of violated the you can meet that all of our
study here's Paul consumption and energy saving in of database system for this is my motherfucking so Montrose at the functionality of changing the operating frequency in in the London and so by injecting application performance performance the information into frequency controller unlisted really it I was save energy saving can be achieved with maintenance of meeting some sort liberal and they
still sometimes require us a question of how it works so this is the graph or follow consumption structuration so it will if the system is idle conventionally fluctuated uh by many factors like temperature or are told power law in the building of over a graph of the so in another major power consumption ivory precise really had to weeks do experiments noting the polymer is very stable period so this graph tools for our consumption is there during relatively stable in full field you to succeed in so when I was doing this experiment I shut off of laboratory every midnight instead experiment in the area and finish finished experiment at 60 years and he went to bed to when everyone is waking up and
this is just the the story all the remaining part of this Stoker at within the fundamental slope of 1 the violation below principles and is basic techniques this is just a sort of basic spherical side or how much time so if you want to a lot more detail you can track this book and in this so I'm going to expression the celery with my practices and experiences of it and it's
move onto principles principles very simple and I and full of defining goals defined called fast so performance evaluation is that must be you the value-oriented sold define the goal of which contributes to use of value and this is very very uh difficult for authors usually sold initial question of perform program might be very value and uh usually subjective full your job is to related it over there and sit Korea and objective called so if you consider a goal maybe you can you already have finished 50 per cent of your job or a 60 per cent of eligible for the remaining pulses keeps the and the goal oriented and designed the procedure toward the goal using appropriate techniques so we want to basic
techniques of performance evaluation there 3 basic techniques of of what they were efficient and fast 1
modelling monitoring is
the expect expressing their performance in a quantitative form so you during the mathematical form the real system is the too complex to wasting of the two fold outlooks formation is the key to understand the mechanism of the target system and to find the important variables and formulate the performance in time of it might account for so this is smoldering and the course of scientific uploads OK I'll let
me give you an example for maybe you know how how this works and the hardest consists of rotating Prada and the header the money magnet here to read downtrodden so this throughput of hard disk drives can be modeled by this easy to from article form so this is the the how many bits of swept by the head in a 2nd so from this small you download that throughput is proportional to the density of the density of the cylinder and the density of outermost sitting there is is the largest and the solar inner cylinder has a lower density and you get
a call from that we measurements so medium and clearly depicts the debate that perform so that other most brother as the largest throughput and in most of the inner product of robots another example the
latency of harvest drive radius your hardest tribes is called consists of the crater the and will additional latency so if you are if you are a lucky enough and data access latency the rotational latency is almost there and but in the most unlucky case you have to wait a proper to roll around the of latency can be expressed in this form and you can call from
the model by measurement this graph shows the made this great is the x axis shows that the distance and y axis to the vacancies so as you can see out there that I don't think distancing increased latency and also this week old lady the band is exact equals to the rotational latency of this how describe our was obtained thinking about being and another
popular modeling techniques to a model came a consists all summer and the we can choose and its behavior is described by I've only of customers and other social science so when you consider CPU performance you can model CPU again 1 of the few a customized CPU instructions and summaries is execution unit in CPU so when you consider storage system as the I O request and so storage controller or individual disks in when you consider the the system customized security and service is that after execute the query execute a process and the
muscles simplest and this present are useful monitoring is and then 1 will allow expression to much did you about this is that this consists of class obvious and 1 infinity Q 4 and we some conditions on reliable later and understand distribution and if you do it as a more mask average response time can be computed as this simple form and it is not so
difficult to extend this to multiple felon servers and using this model we that we can do some interesting and I think of here I've written performance model of order to system we of valuable of all the things of the quake re-creation FIL frequency goes down the throughput decreases and the response time ingredients will you get model of and above all the response time of readers through multiple obvious to a modern and so by by parameterizing left specifically creation and I can form is a response time in this display so and this graph is this graph shows the couple model average response time of Y axis and the slope with them and up right Y axis is the response down on its axis is throughput of system and the measurement results is this graph on the floor of the agency this model freely much is we the triangle of actual measurement so only this simple model you can uh evaluates the performance box of leakage so this so this may damage 1 the is the basis of of frication performance aware energy saving other expression in the form of right the so the monitoring is
a very important step to understand systems maybe good models for life you are good of you and enables you to predict performance and scaling trends and also it helps you to note notice that the incorrect measurement of body is caused by implementation 5 because you know the trend of 10 to the you know the trend of the performance by more than full and that's their system is actually very complex you generally and sometimes more modeling can with rotation about model but this is not about news but small there is also informative because it's it indicates that you're missing something important in monitoring all measurements so that you can you can try and try to include your model and it maybe maybe it's also indicated that the system is too complex so in that case so this is a fine but you should change launch what toward the pole k a full
next technique is measurement and when doing
performance evaluation and many people tends to do measurement fast but measurement without modeling is totally pointless because you cannot understand the results so if you have a model you can validate the result with a model for again understand all this is working for and if they the difference between measurement and the model we can notice that model or majoring in something is wrong so far so of you do not with such we've measurement and do not produce to start with modern
and this is the guide for meaningful measurement full measurement is not just running benchmarking too long there really seems to be considered as also so what role and matrix and the measurement environments and measurement methods and tools then the conduct medium and and analyze the results OK will this is
a very simple fact example story so here assume that I want story systems that can support of some TPS transactions throughput and radical Islam them ideal for many clients so in this case metric is that I outsourced storage systems
so before measurement model performance In this case broke size and concurrency of ideal might be important for performance and also the ideal Juliet attitude depths in wirelessly as in system might be important and the
design of the measurement of environment before think carefully measurement of environment before conducting measurement so then you want to measure of violence of storage system to ensure that I only crispies the early reaching the storage system not just by the main memory and performance measurement is the monitoring of full resources in the system for designing and violent is that designing the flow resources in the system and then a bit
because of the appropriateness of tools find the that performance evaluation is very goal oriented forces therefore do not start we're you're familiar to but appropriate to for their goals and began how about what metric is really measured for example I'll start very famous tool for measuring higher-performance it can measure the number of ideally class issues slum operating system but it cannot measure the number of ideal request actually issued to the individual disks behind the storage controller all the number of of lead right system calls for modification so be careful about what you are here which so we all started
and run a major 10 maybe in the simple case you can run major element with exhaustive parameter values but when you're doing measurement with multiple variables maybe is unrealistic so from experiments with minimal states and you have a good model of model which tell you where you should made next and
after the measurements do not to analyze the later vote so if you a little matches with the model of available to it means that you have a correct understanding of the system the figure and but you we those do not much resume models that something is wrong so she's trying to include model measurement missile J. U.
S. has moved to the simulation simulation
is that for if you don't do not have a target system or if it does not exist in the real 1 unit our simulation is very effective way to evaluate performance or assimilation is also used for the 2 behavior analysis and modelling is uh you Jerry for difficult for monitoring and everything in detail all the system is you just difficult so I'm wondering is down with some of approximation but simulation can do much more of the 2 and in terms
of the delivery system in money into a single simulation technique is ideal the trade in I only play a fast trains the i apart time during the actual execution of our called and literary the ideal pardon of hypothetical devices so in this way you can divide the performance of wireless devices with realistic work this graph
so that only those of ideal play on buyers divisive we use to be see Benjamin in this case and fully training I O ruffled on wireless hypothetical system right and have this and is not this can be checked as it's the floor again tell that how how you can be handled thing every system in very in
simulation is a kind of what-if analysis and based on some of the functions so unrealistic assumptions can easily related to nonsense results beach yeah how about those assumptions and so simulation results other techniques is a bit dangerous so keep in mind that the assumption must be divided into 3 the uh other techniques right modeling or measurement OK I don't know I
cover that's really basic techniques for monitoring measurement and simulation and Michigan it is mostly useful by itself so in order to do deeply we understand the system's performance using multiple techniques and by dating each other is the key OK so what I'm not being
a histogram maybe have speak reviews faster than succeed it tho this OK by 6 hours that are 0 5 1 2 relation to us or performance evaluation is very very fundamental of the T bar capacity of uh rest engineer because it is a key process to understand how the system works on how system achieves performance for and 2 in so because performance evaluations should be contribute to a should contribute to use value of this is very goal oriented process and then procedural performance evaluation should be designed with a basic techniques monitoring measurement and simulation so this is the basic salary of performance evaluation and with and based on this theory your X billions is made develop your own philosophy of performance evaluation without soul mind and that's all for my talk thank you for listening and 1 of the questions that we we would derivative like to the people of the opposite but critics preventing the you of yourself in that part of the most that you need you see 1st of all you need to do this is not what you are and that's very important program and then in in that point I'm not experienced because when you rusty so maybe all you what experienced in so called communicating with business what are the important point is this simplifying things as much as possible full so if you are out there is to it is possible so trying to who you are trying to calm during the actio of economical impact of all of you know of introducing new authorities in was of 100 eventful 3 and so keep keep keep it simple on and so they trying to translate the technical general to from business presence language for the and so you christian also emerge from the root of the sum of the real data that you read in the world and in the things that you will work and that is the really and all this kind of thing and that's that's the that's the bottom line is that you know that there is also lot this is going to this is not something that is obviously precision and yet none of the theory the results of all of what it is it's a little bit higher than that of 1 my language and a lot of people and I have my my my my my my my my that's my no my my my my the Government of my my my my friend and that about all over the world the implications of this is the right to life of the cost of all almost all of if you have uses they had inheritance also on the constraints in which the radius is ready to on the web request pixels that sort of your brain is also 1 of the things you so you don't you're on on building your own search direction that's a lot of our people because you have impact the human all of this is people don't seem to have a new possibility of being called on the cut off the top of a lot of work the loss is that we also see the rest these for you to the fact that the 2 of them there was but it wasn't any questions that no questions at all or whatever think watch