Y2K and Other Disappointing Disasters: How To Create Fizzle

0 views

Formal Metadata

Title
Y2K and Other Disappointing Disasters: How To Create Fizzle
Title of Series
Number of Parts
69
Author
Waterhouse, Heidi
License
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Identifiers
Publisher
Confreaks, LLC
Release Date
2017
Language
English

Content Metadata

Subject Area
Abstract
Risk Reduction is trying to make sure bad things happen as rarely as possible. It's anti-lock brakes and vaccinations and irons that turn off by themselves and all sorts of things that we think of as safety modifications in our life. Harm Mitigation is what we do so that when bad things do happen, they are less catastrophic. Building fire sprinklers and seatbelts and needle exchanges are all about making the consequences of something bad less terrible. This talk is focused on understanding where we can prevent problems and where we can just make them less bad.
Loading...
Server (computing) State of matter Multiplication sign System administrator Patch (Unix) Floppy disk XML Number Revision control Computer hardware Gamma function Firmware Physical system Boss Corporation Remote administration Physicalism Grass (card game) Data mining Arithmetic mean Word Process (computing) Software Gleichverteilung Figurate number Reduction of order
Group action Building State of matter INTEGRAL Code Multiplication sign Time zone Vibration Fraction (mathematics) Array data structure Prediction Video game Strategy game Bit rate Flag Physical system Injektivität Programming paradigm Spacetime Building Software developer Electronic mailing list Instance (computer science) Entire function Virtual machine Process (computing) MiniDisc Condition number Right angle Code Quicksort Cycle (graph theory) Reduction of order Point (geometry) Trail Computer programming Read-only memory Server (computing) Game controller Wage labour Real number Control flow Branch (computer science) RAID Regular graph Code Power (physics) Wave packet Number Bridging (networking) Alphabet (computer science) Utility software Reduction of order Software testing output Subtraction Metropolitan area network Time zone Inheritance (object-oriented programming) Cellular automaton Element (mathematics) State of matter Mathematical analysis Code Wave packet Software Logic Function (mathematics) Finite-state machine Family
Group action Building State of matter Ferry Corsten Multiplication sign Scientific modelling Combinational logic Mereology Software bug Video game Cuboid Position operator Physical system Product (category theory) Process (computing) Moment (mathematics) Constructor (object-oriented programming) Physicalism Bit Proof theory Wave Arithmetic mean Message passing Data storage device Tower Module (mathematics) Hard disk drive Software testing Right angle Quicksort Asynchronous Transfer Mode Laptop Point (geometry) Game controller Variety (linguistics) Drop (liquid) Computer Code Power (physics) Number Computer hardware Software testing Computer-assisted translation Subtraction Inheritance (object-oriented programming) Kälteerzeugung Forcing (mathematics) Analytic set Set (mathematics) System call Software Personal digital assistant Computer network
Axiom of choice Point (geometry) Complex (psychology) Server (computing) Service (economics) State of matter Multiplication sign Real number Combinational logic Mereology Medical imaging Moore's law Internetworking Lipschitz continuity Software testing Physical system Spacetime Inheritance (object-oriented programming) Interactive television Set (mathematics) Connected space Uniform resource locator Message passing Database normalization Software Vertex (graph theory) Software testing Kolmogorov complexity Figurate number Object (grammar) Catastrophism Combinatorics
Point (geometry) Cue sports Web page Complex (psychology) Server (computing) Building Service (economics) Divisor Multiplication sign Combinational logic Water vapor Chaos (cosmogony) Parameter (computer programming) Weight Mereology Fault-tolerant system Power (physics) Number Chaining Causality Bridging (networking) Software testing Traffic reporting Physical system Service (economics) Standard deviation Multiplication Data recovery Structural load Element (mathematics) Constructor (object-oriented programming) Physicalism Digital electronics Flow separation Component-based software engineering Spring (hydrology) Stress (mechanics) Phase transition Software testing Right angle Kolmogorov complexity Diagonal Combinatorics
Trail Database transaction Server (computing) Digital electronics Service (economics) State of matter INTEGRAL Multiplication sign Mereology Event horizon Power (physics) Sign (mathematics) Video game Stress (mechanics) Energy level Flag Elasticity (physics) Automation Software testing Scripting language God Physical system Installation art Product (category theory) Channel capacity Computer Data recovery Structural load Planning Coma Berenices Unit testing Message passing Database normalization Integrated development environment Software Data storage device Stress (mechanics) System programming Data center Speech synthesis Website Software testing
the the the the
the the the alright so
why 2 K and other disappointing disasters verbal warnings and I'm going to talk about and people dying not just like a little the last year's disasters where people die so that's another thing you can handle right after lunch no shame and leaving but I had when you had so many of you were working in software or technology in 1999 I the few the proud and the brave the grizzled it was in December 1999 and the young sysadmin of my acquaintance had just that her boss that their systems would go down state of this bad her job if anything went out of her boss that she did not mean to bother coming back to work he wanted her to stay on the the servers all night and watched them to make sure they had stayed up but she was 20 something in 1 of the drive from the mining of to Minneapolis 4 hours away and good to have ready with all of her fr and she wasn't worried about why 2 K so a lot of you may get cages this is how I use a lock as was actually the modern version the old 1 had to do something along the top of it all back any word on your belt until you got it in your toilet and then you got a better belt and when it went up to go find a people so that you can call the person back that's all you got was a paid with the number and maybe a little tiny this a data about what we have so you go to a club and they're always be some grass looking person hanging on a phone outside the public like this going it went down yeah how no come in and as you'd actually have to hang out of cars to talking to people and they were not actually super convenient in the world located somewhere super busy and it didn't really matter where you not pleased about because you always had to go and fix it in person with your actual figures because remote administration was in the future and we work in the future yet because it was too fast so sysadmin Hanegem much you know it was a good do any good for hours away it's as good as Knuth you know the end of the world did not but she left it way because she knew nothing was going to be rock and she thing was going to you're wrong because she had spent all of 1999 upgrading and hardening system and this didn't mean like a trivial deployed this means here the wiring in a floppy drive so that you could put the floppy in the floppy drive it up every year that was what it meant you had you had a right people and asked them to send you physical pieces of software magazines were full of patches and fixes and alarmism them like this was an equal distribution maybe a little more alarmism and everything we spent a lot of time preparing for why 2 K so I sysadmin had passed always scripts and install the software patent traced back every piece of hardware software firmware weekly where coded cables that we had had the and then it would be even more nightmares that it was Everything traceable at all were tested it she knew nothing was going to go wrong as
we get further from the reality of what it was like to do mitigation why 2 K mitigation we've been forgetting how many of you were in school during why 2 K right you never experienced this is not your problem of we we get further and further from the millions of dollars and person-hours that went into producing our risk utilities did fail but they filled in like july of 1999 in isolation from the grid because they were tested and they figured out how to mitigate and so we did not have a cascading failure of all of our power systems because they knew that it was going to stay at and probably about half to a quarter of the people you know who were in the industry did not go out and party like it was 1999 they stayed in their server rooms they stayed in their rooms they stay nonconference bridges to make sure that the world did not want back then we didn't really have the idea of push updates and we didn't really have the idea of in regular updates like the longer your system stayed up the better it wants if you haven't taken down the hatches something and done wrong he would count the serotonin years with like a point of pride so it was a really different paradigm take everything down to catch the space program which we think of its the at a collection of labor and produce and material and we got a man to the moon we think of that is amazing that tiny fraction of how nightmarish it was to be why 2 K so it's easy now for us to say that it was overblown because nothing happened usually it wasn't that big a deal where we so have about this out it was just it wasn't a real thing well it wasn't a real thing because we worked really hard to make it not a real thing and it's easy to forget the disasters that don't have nobody remembers all the times that they almost back into somebody or they almost ran over a kid in the road it's scary for a while and then you move on all of this why 2 K preparation was risk reduction we knew that there was something that was going to cause a problem and we did everything that we could prevent it from happening we do risk reduction in a lot of ways like we do vaccination so it don't give diseases and we do antilock brakes so we don't hit the kid in the road and we do trained dates some people stop parking on train tracks we understand how to reduce the risk of things and how do we know how we this like what are the essential elements of reducing risk it's impossible to avoid all your wrists you can't do it we make trade-offs with this all the time and those tradeoffs the complicated and sometimes the logical air travel is much safer than driving but will still prefer to drive a vast majority of the time because we feel like we're in control there are lots of subtle things that that we do to reduce risk rather than becoming overwhelmed at the idea of trying to avoid all risk in our lives we can figure out what is in our control and reduce the risk that and we have to decide how much risk we can tolerate in each cell France and a cyclist and I cycle on the roads in the winter in Minneapolis and that's a risk I'm willing to take and people died doing it every year cyclist die because they get it because people don't see that well I try and mitigate that by wearing break clothes but I also carry a lot of life insurance like I've accepted that for this lifestyle I'm just going to have to deal with that in other ways and I die my family needs to be protected that's how securing myself another example I work directly adding documentation to the eyes and there's a risk that I could break a customer's integration if I do something wrong but we have a standard way of preventing that which is to say somebody goes over my commits before we actually push them make sure I haven't of something insanely stupid which only happens once in a while but you really want to avoid this is some standard but we don't usually think of it as the risk reduction strategy there are a lot of risk reduction strategies that we just built into our lives feature flags and development branches test instances and very launches and deployments then just all sorts of things that we think it is best practices so we've ever trace that back to best practice to avoid risk another way to think about reducing your risk is to predict that the state's something can end up I love state machines I the English major most likely to draw a little heart emoji around a state machine I and the formal way into of always yourself could end up in is called a finite state machine what are the transitions and what are the end states and how they get there you can draw this all out for almost every piece of software and more horrible and complicated it is to draw out the more you should be thinking about microservices of because once again a finite-state machine that's like this big it's very hard to secure properly so to use this as as rescue enumerate all the possible state your system could be and what would make that happen and then you can address the transitions that would cause that so here's an example I want somebody to what is that code and I know you can read this I only want them to put in a five-digit ZIP code of 1 extra I don't 1 under the don't want anything invalid so my state is an analysis of what they enter if it is alphabet if it is more than 5 characters if it is less than 5 characters I rejected try again and you're real that of this time you know what it is only 5 numbers will get through my culture those are the only 2 possible so so having figured out what my tutor possible state terror and derive what the risk of somebody doing it wrong is it thinks that it could forget 1 I get added to the list below yeah you also can't use characters you now reduce your risk of invalid shipping and war weird injection mitigation is the flip side of risk reduction harm mitigation doesn't matter until something goes wrong no matter how much we try to avoid missing something is going to go wrong the truth is that bad things do happen all the time this is why we wear seat belts and life jackets and and spot install fire sprinklers and smoke detectors and doors that open outward usually we don't have a house fires in the 1st place but if we do would like to survive so the most concrete example of the harm mitigation that I'm going to talk about is building codes you can also think of seat belts entire mitigation we hope you don't get in a car accident but if we if you do we help you don't go through the windshield stuff is already gone wrong you're already in an accident how can we make that less than rain arrays the assumptions that some of your inexpensive disks are going to fail you just have to have the data distributed across enough just that you can accept that rates but building codes
building codes save lives in 1985 an 8 comma decimal 0 earthquake destroyed Mexico City at least 100 thousand people died Mexico city is built on a lake bed that was sort of kind of building it's really prone to liquefaction and of another quake hit that everything jiggles likable full job is not an ideal substrate for any kind of a seismically active zone and this was the hospital like that was 1 of the better built buildings the whole city of Mexico pretty much look like this this is the way what happened this year it's pretty much the worst picture I could find of what happened only intervening 30 years like Mexico city insisted on rebuilding with stronger better more robust building codes nobody put of seismically unsound buildings because they knew what was going to happen and they had this really fresh vivid memory of thousands and thousands of people died in building collapses so when Mexico got hit by a 7 comma decimal 1 earthquake the modern buildings survived well and it looks like a Mexico upheld Mexico city death toll will be 228 and this is not because Mexico City is smaller if population is 25 million but they even installed on early warning earthquake detector that detects a super low vibration of low-frequency vibrations they get that earlier people Mexico City having 10 2nd warning before they get hit by actual tremors that they can feel Mexico city knows that they are in a bad place and they're just doing as much harm mitigation as they can they're gonna get hit by earthquakes but they can mitigate how bad it idea the last example of building codes is
even a harder for me on your right is Grenville touts 24-story residential building we still don't know how many people died in this fire we haven't been able to get it and find them a lot of things went wrong but 1 of the primary things that went wrong with building codes it had a flammable kept cladding that which the fire up the side of the building defeating the slightly fireproof company box displayed construction the cladding outgassed cyanide when it burden and there was 1 1 evacuation stairway for 24 floors and it was blocked there were refrigerators there were mop buckets there was all sorts of crap in the stairway in a way that would make an American wage because you don't think about it but every 1 of the doors that you get from here to outside will open our work because we have the trauma about pictures of people piled up in front of doors that open in you can maybe think about like Triangle Shirtwaist Factory pictures and how people were blocked by large stores you can think about the ghost ship fire 50 people were not building and 36 of them didn't make it out because there were no fire exits but if you thought about it in yesterday a fire alarm went off you know where to go even if you work showed that there was a fire the 1st thing that happened was that you had the doors and they swung open our work and you have the next set of doors and they swung open our work and emergency that little bit of litigation saves so many lives this other picture this nighttime picture is the sadly me the torch tower in device that have the same kind of cladding the same aluminum of flame of cladding on but it had for exit stairways and its 78 stories tall everyone got out of OK no debts no injuries because of building codes It's not that these didn't both catch fire in a really dramatic way It's that building codes mandate you be able to exit so think about that when you're thinking about what those ha mitigation do we accept failure and we make it and not bad as possible so the 1st step to mitigating is acknowledging that we are never going to be able to fully eliminate rats which is kind of horrible force like as a parent I would love to be able to wrap my children up and make it so that nothing bad ever happened to them but it's not really a a workable technique so all I can do is sort of life proof of they give them bumpers and let them learn what hurts me but once we accept that there's a risk in everything we do we can identify it and figure it out instead of just hiding from so I assume that everything is going to fail you don't know how your product it's going to fail you were missing something if you don't have an idea of what your failure modes are it's going to be really difficult for you make them better for example failure modes can have a lot of different meanings for different people if I leave my phone unlocked in my partner sees my chat history I will get T because it's selling if somebody who with an abuse of a controlling partner leaves their phone of an and partners see their chat history could end up dead but why are you talking to a woman shelter you trying to do something you trying to get away from me your threat model needs to include a variety of people for every failure in a software-as-a-service setting what may seem like a harmless bug around graphics could break of unimportant custom analytics modules or it could be something that nobody uses at all only by using analytics you then to be able to tell what people are actually doing with yourself testers early meeting people who never believed in the happy pass and we need more of them our lights like it's important we do our own testing but it is also important that we hire the professionally paranoia because they have seen all the way that things failed and they're excited about finding more of them and if that's not what drives your life's purpose higher tested do it for you because those people really exist there anything so think about what it is that failure means in your product when we try to mitigate harm we need to think about what it is we're trying to protect them for example nuclear power plants are designed to fail states in the absence of power or positive control the control rods drop and the nuclear reaction stopped that is the idea usually it works when we hear about the time that doesn't but for the most part if your control rods can drop at all they will drop as soon as a power that had on the other hand time log saves the face on oxides to protect people they're designed protect money you not suppose the inside a time lock safe pretty much at any point and so you can't open it without a combination and a set time but and that protects the money and the person because you can worse them into giving a codon have doing good so what we're trying to protect their is what's inside the safe and also with outside the in most computer cases the thing that we're trying to protect his data or state when a laptop runs out of power it shuts down pretty gracefully like all all animals that of power that's actually pretty graceful like a could be so much worse if you've ever had somebody triple desktop cable you know what I'm talking about on that never happens the but in laptops also have this call the whole thing now where if you not often people we all have to save the state of the data that they know about him the moment that they are following like how cool is that somebody's figured out that we care so much about our data that even as the computer is falling to its probable screened and you're still saving whatever you need to to the hard drive where they assess the so you have to figure out what you care about I worked at Microsoft BitLocker for a while and it was really interesting because we realized we even care about the laptops Hardware is easy to replace that access was a real this like the number of executives who leave laptops in text against is really large and the person who is had not lost a lot of that has left of things that I I that's but we say it like that's disposable it's a 2 thousand dollar lab that we don't care is so much less than the data breach of somebody being able to happy into our network with physical access to your computer so we secured it so that it would be easier to accidently wave the data then it was to recover organizing it is is in the nothing is worth somebody else getting into our system that is I risk my again predicting the possible state allows you to understand where you can make a difference by focusing attention on transitions in states they could be altered to be better you really need to think about what is he continued to do continuous improvement on your products to make it more risk tolerance the easiest path in this for software is the kill switch we use this process by redeploying a Nobel but that's painfully slow when you're in the middle of the outgoing prime crisis instead you can always have a transition state of hit the kill switch and then you
could just sort of like walk away like a cat
just off a that image that that was
inevitable and nobody's going here in
that you want to be able to say
turn something off in a hurry without having
to redeploy everything failure is
inevitable we are all going to fail we scale every day we forget to remember to bring on the melt we typos something something happens we fail all the time disasters are a concatenation of failures that add up to a catastrophe we can avoid a disastrous why is predicting state so complicated heart my simplistic examples don't make it seem like it's super super difficult to do this I think even though we have designed for redundancy in failure systems are sufficiently complex that we can't understand all of the interactions we saw this on LinkedIn who here remembers when linking managed to turn on all of the features of what I was the only 1 is not that it was terrible it wasn't at the time that stopped 19 you for a little while they got the stuff sorted out of turning on all of your supper once it is not a thing anybody could have anticipated because there's no reasonable reason do that you have to have that mind set of like what could possibly go wrong no butyou than that in modern software it got really difficult to say I can follow a message from 1 location to another there's of real like Internet magic happens here in the middle especially in software a server you can't be sure which known it's going through you can't be sure which channel is going to look at all like weird that's probably gonna come out the other end this makes testing kind of difficult because you can't build with just the nodes just around parts of the finger choice and you can build with just the sex of the figure tries you have to have both the software and the channels micro services and the connection multiplying combinations growth super complex super rapidly like if you tried to test every possible for a mutation of a tanker object of you quickly run out of computational power this is just to objects in space rotating around a central point imagine what it's like when you
have 60 microservice and they have more than 1 channel each you contest test all of that so you really have to test each of them test might service test channel and then when something goes wrong trace it back and find out where it actually fellow you have to give up on the idea of 100 per cent test coverage to do that not thing I don't think I cent as
coverage is really possible for software-as-a-service certainly not for any kind of complex microservices because so if a disaster is a combination of failures that causes the system to stop working usually in a way that cover the money every disaster that you can think of from natural disasters to engineering disasters has several contributory factors right so American high winds and storm surge of debilitate buildings to take up OK that's bad but it's really bad with in the the power is also out and people can't get water and they can dispose of waste anything on this 1 thing and that's not just that your roof is gone it's on so that you don't have clean water or place to store your of medication for a place to sleep at phase disasters are about a multiplicity of failures 12 hundred miles from here at the end of the navigable Mississippi there's an interstate bridge it is not this interstate bridge this interstate bridge fell into the river it was some pretty unpleasant via Minneapolis right that it was a beautiful spring day and it felt very suddenly there was almost no warning and king people died which is a miraculously no load number because is the 4th busiest bridge in Minneapolis Saint Paul the engineering report says the primary cause of the collapse was the undersized gusset plates at point 5 inches thick contributing to that assignment or construction there was the fact that 2 inches of concrete had been added to the road surface over the years increasing the static load by 20 per cent another factor was the extraordinary weight of construction equipment and material rich resting on the bridge just above its weakest point at the time of collapse there was another 262 tons consisting of water standard vehicles it wasn't just that the gusset plates were too thin although that was the main cause they'd stayed up for 50 years by that point it wasn't just that there was a lot of concrete on it that have been a lot of concrete on 10 years it wasn't just that there were construction vehicles on it it had 3 major research physics is that all of those things came together at 1 time to contribute to a disaster and it was by the grace of over engineering that it stayed out for 50 years because if it had been stacked properly it would never have fallen down and if it had been built shoddily it would have fallen down much sooner it was only when you got this combination of factors that we exceeded engineering parameters to keep it up when we finish our own disasters out of our metaphorical river and piece them back together we find out that it's the same this cluster had a bad shot but that didn't matter until the other cluster blipped through a bunch of traffic over to the 1st cluster and then the shard failed to replicate it's always going to be something like that so the only thing you can do is build in engineering college that allow you to recover from these problems we have to give Netflix Netflix credit for popularizing this idea tool flying chaos to head off disaster causing minor failures and seeing if anybody notice that in if anybody notices your engineering college is not big enough you need a bigger tolerance you need more servers because you set your parameters for Chaos Monkey and it takes servers out and if it takes out of servers that somebody noticed is then either you parameters are not an accurate portrayal or you need to add more servers here are some general concepts that I'm going to go over really quickly but if you want them all on 1 page this it's time to take a picture Here's things because disasters to fizzle into failure use microservices and very loose coupling Chad mentioned this in this talk not pet not even cattle think of them as bacteria the system has to survive the death of any bacterial thing your server doesn't matter it needs to not matter and needs to be irreplaceable element use very loose coupling use internal API so that if something goes down it doesn't chain react in cause other things to come down with it if you're API continued obtain something that's down you will in fact have a problem because now you DBOS yourself which is hidden unpleasant and unfortunately common because you wire to up to actually find chalk Deligne 1 of the common mistakes that we're making mitigation is that systems can't always talk to each other smoothly if there's a problem sometimes take out the messaging too how many of you remember the status page on S 3 was inaccurate the outbreak because 1 of the elements understands the 1st 3 was hosted on history You're messaging about failure and needs to tolerate being part of the failure so if you have a status page
lecture you're not hosting it yourselves status stage Eitel will host for you they are science down and everything but and then he will stop calling your tech support to find out if your speech down and everything is on fire storage is cheap make sure you have your data in more than 1 place because nothing and nowhere is state Delta I am looking at you the other had a profoundly nineties data center in Adelanto and they had when they acquired Northwest Airlines perfectly make with the in Minneapolis and the latter 1 caught on fire which would then find Minneapolis was out of the decommissioned they had decided that it was not worth the cost people redundant data center of and more the the data catching on fire so Jupiter and more than 1 place test your feature in production you cannot have any modern software fully intact it is and possible We're talking billions of transactions and messages and you just can't do that absolutely do your unit test and your integration testing and you're going to test and you're you don't have a test server and I think that I'm saying for the love of God you're gonna have to test in actual production where there's actual traffic no script on earth is going to write you another messages to make it look like but lecture it's just not a thing so roll out your stuff to a few people at a time make sure it goes OK increase your all out as you go because that's going to make it a lot safer to test in production rather than just flipping the switch to humans are the slow fallible need part of almost any system that's a great sign when you're trying to prevent cascade failures of over reaction but when we in our world to react we also need to automate ability to shut things down a circuit breaker in electricity stocks overtures from just circling around and causing fires in the unpleasant things but we can install circuit breakers in our software with kill switches that that hey if somebody is pinning you like 500 times a 2nd about whether or not you're up or down on you just killed FIL life you me to listen to it and out of mating our circuit breakers enter kill switches is going to give us a lot more safety because it's better to shut something down then to have that 1 failure spin out into a giant disaster if you have a run book for how to get a system back up on why haven't you automated like you know the system went down OK that's good you don't necessarily know why and because of the mystical power of how computers work sometimes turning it off and turning it on again make it feel that I would love to tell you that we have solved this problem were actually good at diagnosing why things go down on my whole life percentage of the time turning it off and turning it on at the feel better solidarity fallen over and turn itself off Let's autonomy turning it back on by itself because again we're the slow me part of the system there's no reason somebody has to wake up at 3 in the morning and hit the switch and run around look to turn something off falls over again Sure wake among but like try the 1st time on automated test furloughed and stress and added like it said you can't fully test modern software in a test environment you have to test it under load enough to figure out what your people it is and then calculate beyond that play in the old days and again I'm going to date myself here we have this phenomena called getting sloshed and kind where you would get mention them is extremely popular site and then everybody in the world come look at your site your tiny little site that was used to getting 50 here today and we get that I was in the hundreds of thousands and server would fall over in a dead faint like a guy out there and then they would be angry because but they wanted to see the school thing that slash I and effective steps and we don't have as much trouble with that in like an elastic capacity world but we still need to be testing for slash got level events happening to us because it's easy to mitigate it's easy to prepare for like maybe you need that bottle inbound traffic and nothing you necessarily need that the like stacked up like Amazon at all times but you need to think about what would happen if you got Amazon level track so if stock was too along any rent whether it's of listening to me our expected earlier is it's going to happen make your systems less rigid and more nimble plan for disasters and expect that they will happen to you and greater service gracefully so that in an emergency situation you can still offer something instead of a whole lot of 404 I work for a coal company called lunch darkly we do future flag that the server and if you would like a free T-shirt I'm too lazy to hold them around the country where you can take a picture of this and go to the site and we will send you what thank you Charlie the
Loading...
Feedback

Timings

  643 ms - page object

Version

AV-Portal 3.10.1 (444c3c2f7be8b8a4b766f225e37189cd309f0d7f)
hidden