A/B testing: what your mother never told you

Video thumbnail (Frame 0) Video thumbnail (Frame 3818) Video thumbnail (Frame 6808) Video thumbnail (Frame 9399) Video thumbnail (Frame 12573) Video thumbnail (Frame 15744) Video thumbnail (Frame 20221) Video thumbnail (Frame 21237) Video thumbnail (Frame 32863) Video thumbnail (Frame 41236) Video thumbnail (Frame 49609) Video thumbnail (Frame 56112) Video thumbnail (Frame 58657) Video thumbnail (Frame 59648) Video thumbnail (Frame 60987)
Video in TIB AV-Portal: A/B testing: what your mother never told you

Formal Metadata

A/B testing: what your mother never told you
Title of Series
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date

Content Metadata

Subject Area
People keep hearing about A/B testing, but not a lot of people understand it. Rather than focusing on what your software does, it helps you focus on what your customers do. This talk will introduce some basic concepts of A/B testing, explain some common mistakes people make, and (if I'm lucky), will introduce the first open-source A/B testing module for Perl (I've already written it, but it needs to be renamed and have a better interface)
Crash (computing) Word Meeting/Interview Term (mathematics) Multiplication sign Self-organization Variance Basis <Mathematik> Software testing Computer
Axiom of choice Group action Plastikkarte Bit Parameter (computer programming) Line (geometry) Variable (mathematics) Number Formal language Word Googol Computer animation Software testing Data conversion Resultant Metropolitan area network
Statistics Observational study Confidence interval Multiplication sign Set (mathematics) Mereology Disk read-and-write head Theory Number Product (business) Hypothesis Revision control Spreadsheet Different (Kate Ryan album) Energy level Software testing Information Cellular automaton Sound effect Bit Word Computer animation Website Right angle Quicksort
Trail Dynamical system Statistics Game controller Divisor Confidence interval Multiplication sign Water vapor Mereology Computer font Graph coloring Number Revision control Different (Kate Ryan album) Software testing Circle Mathematical optimization Thumbnail Covering space Area Default (computer science) Programming language Key (cryptography) Weight Graph (mathematics) Word Computer animation Visualization (computer graphics) Personal digital assistant Video game Website Right angle Cycle (graph theory) Resultant
Point (geometry) Computer animation Summierbarkeit Software testing Student's t-test Number
Presentation of a group Randomization Existential quantification Group action Building Context awareness Confidence interval Multiplication sign 1 (number) Set (mathematics) Solid geometry Parameter (computer programming) Disk read-and-write head Web 2.0 Different (Kate Ryan album) Hypermedia Thermal fluctuations Data conversion Position operator Physical system Area Stress (mechanics) Sound effect Special unitary group Bit Peg solitaire Computer Process (computing) Order (biology) Website Summierbarkeit Cycle (graph theory) Whiteboard Quicksort Freeware Arithmetic progression Resultant Point (geometry) Slide rule Overhead (computing) Service (economics) Link (knot theory) Observational study Divisor Online help Mass Regular graph Polarization (waves) Theory Number Power (physics) Product (business) Revision control Well-formed formula Boundary value problem Software testing Punched card Form (programming) Mobile Web Information Weight Variance Line (geometry) Limit (category theory) Word Newsletter Computer animation Software Personal digital assistant Blog Video game Library (computing)
Point (geometry) Group action Multiplication sign Gene cluster Frustration Student's t-test Mereology Theory Field (computer science) Software bug Product (business) Power (physics) Revision control Software testing Selectivity (electronic) Data conversion Condition number Physical system Thumbnail Area Programming paradigm Pattern recognition Meta element Expert system Line (geometry) System call Category of being Process (computing) Invariant (mathematics) Computer animation Personal digital assistant Search engine (computing) Blog Universe (mathematics) Website Figurate number Quicksort Family Reference data Resultant
Point (geometry) Computer virus Dataflow Statistics Observational study Interior (topology) Code Multiplication sign 1 (number) Set (mathematics) Web browser Student's t-test Rule of inference Number Formal language Product (business) Revision control Facebook Robotics Hypermedia Different (Kate Ryan album) Profil (magazine) Videoconferencing Software cracking Software testing Data conversion Physical system Area Enterprise architecture Focus (optics) Stress (mechanics) Sound effect Bit Basis <Mathematik> Search engine (computing) Personal digital assistant Universe (mathematics) Vertex (graph theory) Website Whiteboard Quicksort Resultant Writing
Computer animation Information retrieval Variance Software testing Table (information)
Dependent and independent variables Computer animation Data conversion Quicksort Theory
here it it it
it it you know it but we thank you thank you but they allow to get all the time of companies saying all organ make lots of money based guesses about what works by the Digg . com is 1 the most famous examples of how a crash and burn with that so you may refer the that all been terms yet I don't know them so they take a test works can like this antagonist skip a lot of the details don't ahead to the further so basically you're currently here's what we technical basis and then you have to new behavior and try out the we call very and you have 1 or more of the of the variance in experience but for the purposes of this we're just have 1 very making things lots of words and move along here and you randomly assign customers to either the base your original behavior for the variance the human data and the question is what we actually
tested with the testing basically anything which has a yes or a no did they click the biobanking or not they signed up for them is whether or not they agree to receive spam from you or not that's where the testing does very much simple you know yes or no choices so please keep that in mind when you're looking at in implementing the testing some of
them for actually get into a little bit of marketing so we should talk about 1st see understand customers who so this is what we call the conversion funnel now I made up these numbers just to make the man really easy but this is a very important concept the testing the idea is you get lots of customers sitting a thing and then you get fewer and fewer customers because of the way the words following them down and as long as the million people see you add words out on google for example we assume 3 % covers a step that's a number that means 30 thousand people research with Victor words will by analyzing the partners involved we groups of variables and then I realized in the line of the and something a really have 27 people who did not in the car and convincing by giving a limited number of the credit card number through so we have 1 person died but 1st language in every step now think about what it cost 10 cents for parameters click here's cost you 3 thousand year old yet 1 customer that's at all costs subtracted telecasters that's a lot more because the Atlantic build whatever it is you're looking at the museum you how specific you merge and Pontil long and so generally what they're testing you stop start at the top of the conversion funnel picketed a lot more to it and get your results in faster but were you going that material so we will just have the finally the so have just
a curiosity which 1 of those do you think will generate more clicks on a web site and the book you but anyway this will be wrong in the editing welcome yes you have an ideologue to go in the of the reason why you're wrong is you have a different customers you have different parts you have different websites you have people who just have different things that you don't know in advance which is going to work so I don't company which decided from horizontal scroll bars but there may be test because the e-commerce site that's a whole idea they made a lot of money off of it and I am not going to detail as to why but you don't know in advance so don't prejudge what is and is not going to work so let's try this again this is real 1 so whenever we put another visitors in our experiments we see a lot of the other about a little bit about what was 125 versions of the original version of of which 1 performed better with the use of ICT was set the license the use of a warehouse radioactive onto the we don't know yet and also your wife sorry try again and again we get roughly 20 thousand visitors misleading more with this time but now it's no more convergence and blue 150 time reverses the hydrogens or the the problem we have for some numbers didn't quite work out this over and the better is because what most you can't be try words to success in your head I hope that and I find remember so and the way it
works out the 1st set of numbers you apple we call 95 per cent confidence level that is the per cent chance that the null hypothesis that point what another study that said were 95 % comfortable these numbers are correct more or less that's been only testing on the topic of ignoring information the basically because website right here this is on Slideshare by the way you can download a spreadsheet used to generate this 1 is the result of different because I made them easier to read to the relative like for this like that there they but effectively and 95 per cent confidence the 1st numbers women 51 % cancer that was better In other words you're flipping a coin with the 2nd set of numbers and I found that we have a nice for sentence that lose outperforming red with those numbers what I mean is that you will never 99 confidence in this summer's going up it only 19 century numbers there's is like brought something like 85 per sentences that and so this actually gives you the basic idea of how this works in a b testing we use statistics to analyze numbers come up with percentages do more let's say we think this is going to perform better than the that but because of taxes cells more product so please keep that in mind you start selling a lot more of a given product Brady testing on way the of less of another product but also the whole sort on increased more profit at something else you have to pay attention to you so side effects and there's also a lot facts it doesn't really doesn't tell you about for the most part that's not an issue if you apply A B testing is properly theory making a lot more money lower assuming you have a viable business and the used food emporium or something like that yeah those the you might so this question for which hurricanes levels
better you'll be confident that the results of it is not in a better than 90 % there would have an opinion off OK so I coupled to question so far and where it is and then in the US in my say that they were aiming for dynamics and confidence but we know from the number of people we have a stickler experiment that that according to but just circle sales by about 10 year old they're it takes a lot longer to get an idea that confidence in something you can do a lot more traffic for a 10 year old age you don't care you're wasting time going in and out of this right if you feel comfortable like a visual website optimizer has a default the per cent confidence level in the results if you are going to be shifting your revenue by 100 thousand euros today I would not feel comfortable in me and others the cover very directly tied into the you know the monetization and possibly other factors so don't just say this because we have to have stick with the yeah how am I doing in time by of the that that undergoes goes up from 0 well defined in the thought of Slovenian and where is our so this is the this is life cycle of the testing and it never ever ever stops at doing graphs up here you create an experiment that endorses blue buttons note males versus some they'll small over large from greengrocers things for things that were not think that what Nicaragua experiments you don't know that ice there is also a customer comes as he reference you crack the performance over time and then you choose the best 1 the for example no idea goes some details on the steps 1st of
all this this is the testing will be born of out the experiments you know what 1 difference per experiment with you have this button and that button then there's 7 differences you know maybe the size of font color whatever the NEC difference in data you don't know which difference actually clause that difference in behavior of the sentence so only 1 different parts per hour for very if you actually have several variants but again it should only be 1 thing interesting differences you so if you have a red and you can try a among the great but in a white cotton whatever in the same experiment if your prior larger size that would be a different experiment because of experiments like that you probably don't wanna run them at the same time having people know the 10 AB Testing the washer currently testing conflicting experiments 1 person finds water thumbnails another person tried moving thumbnails running those at the same time not very effective so only 1 version of the control we're actually was to have an annual that's what customer comes your website you choose whether or not they're eligible for the experiment that's a key thing many people roll the dice and then they decide whether of some experiment they wanted tracking people in their statistics were not eligible for the experiment for example let's say you want to show a US special promo for all customers on your side who spent more than 10 thousand EUR last year and I say that's only 1 person your customers if you put all of the customers in this experiment but then use up but they're only showing this amount the special polo in the experiment to that 1 cent of eligible for you actually did not 199 . 5 per cent of people not seeing that very but only 5 saying and it turns out this is statistically valid but it will take much longer to get results on this so instead you only want to you want you to know customers sellable for the test are they in the top 1 % and then you roll it eyes to see whether or not the tract whether they're in the basin very you only track the cuss words were over In statistics were not the goal of the certain for weighting factors on our excellent tools out there available tracking particularly about some programming language that's actually have tools available for use by Sir writing 1 myself I'm not releasing because interface bad and will make your life worse so onerous on I use the best afterwards you you craters there seemed which version area see intractable for that you can choose which 1 performs best patients is very very important here maybe people get in case of beginning and they screw things up earlier graduates from your wife
and the keep point over and in at this point you know 99 point 9 per cent of what everyone else knows about the test but there's a lot more issues all so far going further than 1 any questions yes there's a of the new students you the if they don't know what the problem might use of that vertical and what they can do about this that you that can think because they like the sum the
and in the that idea of mole of like if you read more and more number of questions that I mentioned the by and it does make it harder to figure out what's the best what happens over
time as the result of some sort of much of the Jews in the book of the of the you don't want to the most that's not me new OK and and things of that and there's going to be so good that you go about and you need to what happens for many companies that actually is when they have a previous experiment like a red blue button experiment and they decide to flip the switch and what will be the permanent behavior what they will often do is leave that experiment and the and and then months later the decided to change something in that area it might be wrong previous experience just say of impact that make sense of the world OK so we talk about how choosing the best this something which a lot of people this is very strong you right off the bat 1st many times people forget to ignore the 1st business day the running an experiment are getting a lot of traffic medimmune excited because the variance is pulling a lot more money than the a lot of random fluctuations particularly the beginning of a test you don't get overly excited in fact it's there so maybe testing tools that will allow you to view the results until you have enough results in just you know get screwed up that can also be problematic because of the results on 1 variant are 0 conversion all the way across the board that you'd like to know something like that or something else goes disastrously wrong you would like to know something like that I recommend running every AB test at least of all business cycle so it was a cycle lasts about a week anyone stochas on Monday and by Thursday afternoon the new version is statistically the statistically significantly better than the old version of the library were coming the weekend working get all these extra sales let's go ahead and its neighbors and you're getting impatient and instruct because what happens is beta customers over a weekend for full business cycle that we might be significantly different these might no longer be people were just browsing here site from the work computer when not allowed to be like the people at Holmes said was he's looking inside by whatever it is they're often with different you have a lot of work at their behavior might be significantly different so you this progress over the whole business cycle just to see how it performs so you don't miss some useful behavior which is likely to happen on a Sunday nite or something like that and that you're wrong very wrong usually with a test remember the confidence levels were 95 per cent confidence that this is actually going to improve things don't stress about the but if you probably most of the time you'll get it right we make mistakes all the time every day and we already know that yes it is entitled power is a set of users might not have cycles but basically if you have had a customer data like media traffic peaks at every Thursday to that and then you know on every other Thursday the you know something like that then you let it run like that now this this is kind of problematic having run over business cycle because if your tax accountant in the United States the business speak here on April of every year having test waiting for an entire year may not be appropriate but can be just and that this that help explain where there's a cycle of for the very wary of of false positives is excellent lottery uh and the present is from the from booking that working for a long time but she actually a simple software in there that's the only all you might compose sitting in this presentation where where he showed how you can run a about to experience which know the basal area are people not 1 of my Mrs. statistically significantly better than the other if you don't want for long enough because it is possible you will have a random fluctuations early on in your testing before you should be significant which are enough to throw in the significance and you might call positive and say all we've got something which is better than the old version so let's go and Solitaire but you don't yet know so the trick behind this is when you're starting again going from a blog post which is a little bit more and that explains how our work this out when you're running experiment you decide beforehand we need this number of customers before we feel comfortable saying this is statistically significant and even if it becomes too significant before that you ignore it anyway until runs so full run at least for business cycle no number of customers you said has to be in that experiment that make sense yes the union of over the time yes but the longer it runs so if I go to point 10 times you're surprised becomes a has 7 times by that 100 times that comes up heads 70 times something's probably wrong if I knew a thousand times you know something about the longer the spirit runs the more likely it is fluctuation versus the randomly even up unless there's something it will send them to describe the setting sense all in the house and you know that's a arbitrary boundary lines that the link to another post about vitamins nanoscale here that's a lot of interesting regularities in the mathematical formula you can use to calculate the number of visitors you need to have for particular test to feel comfortable with being statistically significant and be meaningfully statistically significant of boys everything you want comfortable so in the US before it on that so the Sun's limitations existing loving about it just because customers over the red button you don't know why maybe testing will never tell you why your expertise at this point is no longer required for making something this can be better for saying I think layout better limited ColorLayout here expertise for interpreting the results of what your customers say is actually better you know the better and more customers do that's what you want because we need to focus on customers not talk about that after site have you could have side effects may be casting your eyes you might have a beautiful new systems for your pulling all this extra data out of of you you know SQL system to appended to search results of the experiment and they're looking to buy british usable get some accessories going along that it might be there we see 1 difference there but you also have this low overhead of hitting Europe North fuel system that might have other side effects in a system that may settings darkness show you so it's not of fire and forget the solution you do need to think about what's going on with the and it also doesn't tell you anything about customer loyalty over time you went to a great job of you know driving traffic hurting everything's going and your customers how we're media have you read it fails dramatically but your 1st customer service is awful this could backfire that isn't
punch line I have a lot of this formula is again you answer In our view of of the black and this is the ultimate up in the air but interesting to you you so that means that the 17 minutes at the rate of light and only a few slides left so that it's about this is the lowest in all of my life I like what they do mostly on top of these guys are actually very sharp and what they've done is they've built a tool which the masses can actually use and it's fairly simple their limitations in school they're aware of limitations target limitations across the using the tool I don't have any other option than out there basically on some case studies on the side of his case studies all over the web on talks about 60 per cent increase in test that will dramatically lead to increasing by buying and always solve God's going to put this in a party in the form of an increase in the I knew that the factors that increase in downloads there's all sorts of things you can actually test new something's attested in these case studies so we're talking about your ad copy on placement of polar action buttons pop-up forms all sorts of things we might very well be rude aside for for a newsletter and that's the sister placing a big increase in the number of newsletter sign-ups here and in that they actually were basically is having some form is that a newsletter they had a pair writing this is why you want to sign up for an insulin and the getting started how do you get started in 80 thousand from what people wonder about you know the vendor solutions for your building in house solutions up I recommend most people know the minutes solution develops an optimizer of mobile if they start shifting around their products all the time and because they are there and they now there there and since being others optimize later number of others basically the quick and easy to get started with just seeing feel for how this works if irrespective day building their own in-house system argument concerning the testing or you might go wrong which is that I did this presentation to a company in Paris and so the presentation was done they shut down the justice system because it been springs purposeful solid so I the solution 1st once you know once you feel comfortable with then rolling your own in-house systems great what you do they're going for quick wins is important not obvious stuff at the top your conversion funnel would get more traffic you can get results faster changing the size of the headlines removing your headlines or whatever you wanna do the stuff of making your pricing information were usable some brainstorming things you can change this really frees you up when you're doing the testing because now said no know those weights have to be in order why not try you are free try something never tried before I'm also going to be a test this is really important once you get another by running a just as we have basic very exact behavior I knew what was over time so any it will show you how you will get of statistically significant fluctuations from time to time but be able to make and in the out of the also after a while if you continuously have 60 per cent and 40 per cent in 1 group or the other if the splits not even or if they're behaving differently to reproduce version in 1 for exact words and the other over a few months you know from a to the testing set up so running in a test for both sides needed for the same overtime tells you a lot higher in the testing set of this work and if I was going to have you need to know that the of not just in but while I think it might be a good thing because they all the cells in and broken because we obviously have no or only a few people here and you wish I don't really quickly because the and the side you need like a test run for a while because you will have random fluctuations people I mean I'm all you know little point in time you might come up so that some of us and they have a lot of you so you you know the company's business cycle this understanding are business comes in the form of traffic of wherever there is no difference in the theory of that occurred time you need the city's out of date by the payroll processing firms 22 weeks months apart and you know the 1 whom you might need to use cycle is that they and you get lots and lots of wrap it from the sum of of so I can't tell you what your business cycle is you need lots of data from the business cycle the fact that 1 probably longer because it's not hurting because there no difference in behavior so those ones for a long time for a long time when compared to the rest of the tests just because they like to make sure the owner of the system the Working by the longer
because the more variants you have more useful if you have a thousand people if you only have 2 variants about 518 it poses significant if you have and innovation invariance and you have to have 50 people in the universe harder to get this this is the reason the so the occasion is a key point of this yes I yes the trick we did the question was in your local experiments at the same time yes you get what you need to do is you need the wearable in those experiments on a conflict have you have someone who modifying CSS cheating so far a sort of running the experiment a lot of ideas that came up and they don't realize that running conflicting experiment 1 of overseeing all the experiments looking at them at the example I always give his someone is trying to make the thumbnails larger and the person and remove the thumbnails that's a problem but if you're working in groups you may not be aware of that hopefully people to be restricted to the area they're working on so like 1 the back and as using the working of a meal stuff but never the they test theory cannot conflict with anything because tend data as being next thing for no but there anything with the there doesn't we also known as what happens after the really interesting blog entry by a designer all undesiring went to Google and that really takes off our any test everything in 1 time when you're testing 50 shades moves in the opposite case something the show over 60 cents a little on the website of 1 of the better frustrated because she wanted to do a better job of designing making pretty things that altogether slice we wanna do the job of selling products because we're a company article the smaller of you want your review your mortgage for you wanna not building from it is nice to get trapped as saying I know this is going to be better but you find out that it's not so whenever started the testing whether it 1st test I ran out of work of based on under a particular condition the search engine throwing all search results well that's about it very clearly about it was not documented by my back to the you know the source and so long as there was nothing in there thoroughly by OK so screwed up but as a general I don't trust 180 testing and just fix bugs and it's about their around the bugs in the test but I push them out there because just because some data but he doesn't mean that it's going to be adversely impacted customers so I wrap this up a test with the out there in behavior was manifestly better in my opinion because you were for search results and it performs much worse so this is simply an undocumented features were looked like but but in fact we did want for only the search results and for some reason someone in the past found out they were born for meta reference data for the recognition in the covered in resource so this is the 1st but I was convinced that I was tried by you by our from other people come in no of recall layout is much better than two-column layout what you doing I have been doing this for 10 years you don't care about that the fact that sort of thing you have to work around because what you are saying it's not we're trying to answer a call that desire is always because you try everything because if I don't get customers respond to better that we want to use the expertise comes in and evaluating the results afterward testing can tell you why we need the experts in the field of a gas and figure out what all artistic problems they be tested it into the brain can form a favorite red lines you roll the dice all were so low but this person but you catch the rat but that that the earlier story result of non-random selection power that is in 1 group and returning users another remember where where the other other no no no that's not statistically valid you need to randomly assign people to the what you can do before and is just 1 other eligible for an experiment but once you decide what the eligible randomly assigned to 1 of my forgetting selection at the 1st technical matters in the vicinity of your site is suitably but every time they come back in using the but don't start switching back and forth all the time of the test data because you will know how it actually impact that so the company paradigm that giving students but actually destroyed during the test users that carrying out right but Everex's customers but because the from the customer's experience there actually a C so they kept randomly showing results and then encoded problem I've seen this a lot of don't frame-based based and there's an experiment from 3 years ago which is probably no longer valid clusters of the system after a while so this is the frustration you were not part of my family generally of profit virtually the 1st conversion just because solemn works out of your property more maybe testing is not solvable but is to some of the most
experience they'll they don't matter at the beginning you often get a number of quick wins but after a while after the quick obvious ones go away and your starting to you micro testing of smaller things where C is you here's a baseline air fail lose little money or they'll lose all ammonia films limit its role as a way so sitting down here here there also only get 1 which in principle you keep that and I just want to let the fail fail fail follows way of themselves and you just keep gradually increasing how you're doing some also will fail do not get discouraged by that and there's actually quite a lot of money on 1st when I 1st started doing because all these ideas and really great stuff so you see it on a very quickly the value of my expertise and all experience earlier this is the thing I mentioned before you have a million customers coming here website and only 1 set of are eligible for an experiment that put all million in their societies are eligible for the experience that only take that 1 per cent for eligible then you roll venues Ackerman role lies in the ground in the in the in the tested by this that's what I mentioned that but I saw the search engine were referring way results and it turned out that 1 was not about borders on a recent case study areas right where all a conversion Data people actually spend money they removed the Facebook and other social media like buttons and they significant rising version now follows a general rule that once again I'll point fighting you don't wanna distracted them with other things to think about we only know that but not replicated people say the test on your site because the cost behavior will be different yesterday like this on the basis of a horizontal scrollbar at work so it was that they were shot back on the was so upset about the world also robot this revisiting and revisiting this is where expertise comes in and what he saw after a while it's actually a side effect of scroll bar that had been noticed before so he reverted back to a vertical scroll bar kept the side effect and increase or the idea so have freaks us stupid ideas you know the cracks imperial find out you also don't require roll-outs for this you need have maybe testing control panel so it's several in the the code to implement the test you roll the AB test and then inner control panel you can say as far off that makes it much much easier and it's just goes disastrously wrong if the 2 of them children so was said and provoked there's a company working within the US was considering opens sourcing there's but it needs a lot of work and I don't know if that's going to get pushed forward uh so I've since like in movies got 1 rails there and it usually have a title of the article and approved musical rock of all of this but I think that's because we're remove bit obsessed with various offer testing on the focus of our customers with so and I said I'm I'm working on some which is calculates statistics but it turned out to the numbers right the universe so it's not me so the Moon and all the way through the use of from the ones you there are also give you all the way to write it off and then don't know don't stress about that basically people have come up with ideas all the time particularly when you get the idea of the testing what happens is you now have the freedom to think about so we're going to talk about the writing of the Board's classifiers you can this or that what I was writing screenplays I was coming from the every day it was not to work on that sort of coming up with an idea that because got in the house and that's what happens with 1 right there and you think that is because they get used to have people right out of the board because history what is really interesting ideas for
experiments will start the flow but you have certain things that I so segmentation uh along the problems like I have something to think about it so you can train customers offered a radically differently so that some people will not expose the student customers will say it has only through return of the mistake you might be aroused miracle you might have like a virus across the board increasing conversion in a pretty happy with that but then when you look at your browser you discover all the agent is a working terribly well that they might actually important segment of your customers because all versions of ID tend to be older people were often retired or produce systems often have more money to spend so as a result of those little niche areas of your market might actually a little more do do so so accept language to be important if you have a lot of enterprise start your test of the because no 1 version of the marvelously well and then it would be except language and you find out that no supply terribly well in Spanish know laugh it doesn't run over the video so that's sort of thing that you can easily miss it because looking at the big picture time of day people might behave differently different kinds of data so do I have different behavior of former product that of President the sort of thing you start thinking about how you can be id of the segmentation and we're all that I said you're watching people for a while so just looking at the different work profiles were dealing we also see really more of
those some things on the whole the multi-armed bandit with a little research on this basically entire benefits from a b testing but all of 40 the you put all the variance in the the bandit retrieval thing and as it was long it will automatically calculate what works best when automatically so that it's much much nicer than so much money like table you really
bad usually good by unmodified customer behavior from expert opinion I don't care how we do our customers of on the and that's really reverse comes from the course of the customers was not use I realize that this is Adams
Slideshare . yes I suppose so I believe and then the evidence so yes but would that ever request that sort of thing going on and on and on the left side of the of the of the of the of the of the theory about stuff that's not making any money you don't care about you know you hear that you made no and how is it going to fit in the conversion funnel to anything which doesn't have a yes or no response that you care to come up with a theory of gratitude to the thank