Data Corruption: Stop the Evil Tribbles

Video in TIB AV-Portal: Data Corruption: Stop the Evil Tribbles

Formal Metadata

Data Corruption: Stop the Evil Tribbles
Title of Series
Part Number
Number of Parts
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Bad data breeds like Tribbles. Coddle even one bit of it, and your entire database will fill up with junk. And it's got so many causes! Weird user input. Data races under load. Changing business needs. We can't fully prevent data corruption, so how can we recover from it? In this talk, you'll learn how to fix bad data at every level of your system. You'll learn UX techniques for incremental, mistake-reducing input. You'll get a rubric for validation design that accomodates new features. And you'll learn auditing techniques to catch bad data early, before your database fills up with evil Tribbles.
INTEGRAL Gender Forcing (mathematics) Execution unit Content (media) Data storage device 1 (number) Survival analysis Mereology Disk read-and-write head Food energy Public key certificate Frame problem Degree (graph theory) Inference Medical imaging Entropie <Informationstheorie> Representation (politics) Video game Right angle Booting Data integrity Physical system
Dataflow Concurrency (computer science) INTEGRAL Euler angles Real number Source code Set (mathematics) Software bug Neuroinformatik Wave packet Product (business) Mathematics Causality Data structure Codierung <Programmierung> Computer architecture Physical system Dependent and independent variables Software developer Structural load Database Database transaction Data model Computer animation Personal digital assistant Pattern language Right angle
Existence Service (economics) Information Inheritance (object-oriented programming) INTEGRAL Mathematical analysis Sound effect Theory Product (business) Data stream Word Data model Process (computing) Computer animation Personal digital assistant Endliche Modelltheorie Physical system
Complex (psychology) Trail Statistics Code INTEGRAL Execution unit Virtual machine 1 (number) Event horizon Neuroinformatik Product (business) Mathematics Methodenbank Term (mathematics) Operator (mathematics) Energy level Electronic visual display Cuboid Endliche Modelltheorie Physical system Collaborationism Information Interface (computing) Projective plane Flow separation Computational complexity theory Computer animation
Point (geometry) Validity (statistics) Multiplication sign Execution unit 1 (number) Set (mathematics) Control flow Product (business) Permutation Human migration Arithmetic mean Sign (mathematics) Process (computing) Cross-correlation Computer animation Bit rate Cuboid Office suite Endliche Modelltheorie Information security
Word Process (computing) Computer animation Interactive television
Building Service (economics) Overhead (computing) Multiplication sign Direction (geometry) Data recovery Combinational logic Lemma (mathematics) Set (mathematics) Bookmark (World Wide Web) Latent heat Mechanism design Single-precision floating-point format Data conversion Error message Physical system Validity (statistics) Interactive television Coordinate system Bit Data management Digital photography Arithmetic mean Process (computing) Computer animation Software Telecommunication Self-organization Normal (geometry) Pattern language
Slide rule Validity (statistics) INTEGRAL State of matter Multiplication sign Projective plane Electronic mailing list Sheaf (mathematics) Rule of inference Computer animation Representation (politics) Endliche Modelltheorie Lie group
Axiom of choice Complex (psychology) Validity (statistics) Software developer Multiplication sign Mathematical analysis Usability Computer animation Bit rate Software Factory (trading post) Mixed reality Cuboid Data conversion Physical system
Context awareness Validity (statistics) State of matter Real number Structural load Similarity (geometry) Bit Usability Cognition Process (computing) Computer animation Endliche Modelltheorie Object (grammar)
Context awareness Greatest element Group action Building INTEGRAL Code Multiplication sign View (database) Execution unit Set (mathematics) Water vapor Mereology Neuroinformatik Software bug Programmer (hardware) Sign (mathematics) Spherical cap Single-precision floating-point format Endliche Modelltheorie Error message Partition (number theory) Social class Physical system Identity management Simultaneous localization and mapping Structural load Feedback Moment (mathematics) Database transaction Bit Connected space Arithmetic mean Process (computing) Order (biology) Normal (geometry) Summierbarkeit Pattern language Right angle Data integrity Resultant Purchasing Point (geometry) Trail Dataflow Server (computing) Service (economics) Real number Continuum hypothesis Similarity (geometry) Power (physics) Product (business) Goodness of fit Crash (computing) Theorem Default (computer science) Shift operator Validity (statistics) Consistency Weight Database Directory service System call Computer animation Software Personal digital assistant Blog Mixed reality Network topology Object (grammar) Discrepancy theory
Collaborationism Context awareness Computer animation Telecommunication Structural load Multiplication sign Bit Pattern language Cartesian coordinate system Neuroinformatik Product (business) Fingerprint
Area Point (geometry) Slide rule Group action Multiplication sign Software developer Mathematical analysis Similarity (geometry) Coma Berenices Computer programming Programmer (hardware) Type theory Mechanism design Computer animation Energy level Office suite Computer-assisted translation
Point (geometry) Axiom of choice NP-hard Service (economics) Sequel Code Equaliser (mathematics) Multiplication sign Mereology Rule of inference Mechanism design Core dump Physical system Scripting language Default (computer science) Arm Scaling (geometry) Validity (statistics) Military base Feedback Database Database transaction Software maintenance Process (computing) Query language Calculation Normal (geometry) Object (grammar) Escape character Table (information) Data integrity Resultant Row (database)
to the end and posts and the folks I'm that's available and welcome to get integrity in living systems I'm no ordinarily I'd like to launch right in the talk content here without a lot of and that's what you like you by markers keynote just now really hit home for me and I wanted to follow up on that obviously a white woman in his life and these are really different experiences I'm I don't wanna collapse the subtleties comes a lot of ways he frame survival were the ways like my head I I often team experience getting intact for nontraditional background as like this stupid simple habitat the attack manic fixing girl thing on the and energy units is always safe to paint yourself as as an eccentric genius on the material for a 2nd a decade ago when I was getting attacked I was just some shared with no college degree who just washed out of an ox clear because the sudden onset chronic pain disorder the and boot camps were not even thing at so I couldn't even get like a certification that light as in this on my own and that was terrifying and when alive that survival lessons and and this is relevant because there are some places in the store where I talked about places that I use the kind of self register I want you all to remember when you're hearing about those parts that I was only able to properly grow out of that after I wasn't the only woman in the room anymore after was so lonely that the only way I could move from surviving by being a jerk into new ones and kindness and the actual success they bring was not have to carry this torch of being the only representative my gender there I anyway of back to data integrity another stock another name I'm used to called it better corruption stop the evil triples I I did change it because I was kind of hoping infer talk ideas but that reason would imply carrying about my personal dignity I'm and let's look at this example find photometer part why don't waste I don't worry instantly there plenty more humorous Star Trek images ahead I you know they must be to risk because that's how I'm calling them right now I anyway I change in the talk because imposed this frame of bad data being evil invading forces from outside I'm literally everything else my proposal was that how productive reference the and
we visualize their data as this mess the clouds up or otherwise pristine database but we do 2 things 1st off we pretend that there is such a thing as a pristine database or a pristine anything when we're comparing I material and most mayors workin
codebases architectures etc. The look a lot more like this I'm and 2nd reason is that
trains us to think of a situation adversarially and we think of this as the triples of versus the crown I were making a world of heroes and villains we're making a world where heroic developers are fighting that data i is really easy from their there to think of ourselves as the heroic developers fighting the sources of that that data the that is the 9 when it is computers that we're fighting but this attitude quickly turns into thinking of our users as the enemy or even worse thinking of art teammates as the enemy it is super easy to get self-righteous about that integrity she's but once you start doing that we lose our chance to solve the actual problem when you look at the actual that integrity issues are dealt with product changes are actually the usual regards I or maybe miscommunication between genes others also case right where people don't use 1 and that and heavy patterns like transactions real foundations I'm at the end of talk more about this later but I just like to dive in right now and say no this is not your responsibility like you think there's but if we're going blameless when looking every causes we note that when people forget or even consciously skip I'm the at integrity patterns on it's because the could the working is architected in a way that actively discourages the use but sometimes the and you're going to encounter very coherent sets I'm like there could be but impose prayers that emerges at 3 AM on the full moon of a elite year and somehow you data models to vulnerable that I but In owning just Escobar women just not covered at and above on the 94 Cyprus and face on which is that your team culture encode structure are creating a situation which dedicated on a special like designing systems to be resilient against common issues like the product change or an engineer made a mistake because they were tough thing out mild flow I'm like designing assistance resilient against that let us also practically detecting cracked an awful lot of dead integrity bugs that stem from the harder stuff like concurrency under load so
no other country example hangar understanding offer I'm going to tell stories this swinging loosely based on a job I had a few years ago in e-commerce company other details are not going to effect I would pretend this is me changing is the innocent but it was a few years ago and let's theory I forgot was
something monorails model of it was old enough big enough and entangle enough to deserve that name on my team was working a module inside of the took returned goods and ship them back to their vendors for it was not tremendously complicated system in of itself in the large analysis pretty complex but in fact well neither the y nor the underlying data model looks exactly like this you can actually for a pretty good model by looking at this I the you couldn't go let you go a long way by assuming that the only 3 things that returned the vendor cared about were the product its founder and weather then shipped back course no matter how few things your system cares about what the world will come up with a way to only give the partial data they argued case sometimes the incoming data stream would not contain vendor information spoiler it is super hardship things back to the vendor if she not know what that under its the the In other words turned the Vendor Module had under the new article on now that we were returning things the offenders the upstream system needed to guarantee that they were caught we had an issue but the issue was created by the RGB models existence not by any inherent issues at the upstream because the product change the data that had service systems needs perfectly fine the day before was suddenly invalid and this is pretty common then integrity issues the cost of the
product level are often best also solve the problem of we could decide of the out-group like done machine learning or something to identify all these missing vendors i'm but what we actually the feature was a small requirements change any piece of you I small requirements change just don't display the units that are marked for RTD but don't have vendors enlightening RTD display I'm and these pieces you I if we don't know the vendor or something else that the unit on disambiguation interface like kind of a virtual of 1 of the desk that the warehouse workers check things on so the supervisors can guilt at the I am which is what more efficient than like a more complicated more technical solutions and they take ways we can get the story on 1st off scope thereby whether can move forward on closer related validating models in its absolute way is not going to complex enough for complex business operations or separate flexible for complex this operations on so we need to be super careful in how we define a product at any given stage so that we can actually figure out what's relevant ones last thing don't over think that correction and we do not usually need to do some complicated magic to derive information we suddenly need information we have we can just good people and say OK tell me this new and I'm in a little softer system users are as important as code and often in humans are much better at solving these kinds of issues and computers we approach that integrity problems in a spirit of collaboration with our products and our user base intractable problems become track that's a so product changes Ch now earlier i was saying that eventually we found the solution that was uh spoken down in creating a disambiguation face there's a reason I was saying eventually but there was some perhaps some miscommunication on this project before we got the eventually 1 against this year's I forget the exact cost events here I by in general I might seem assumed that upstream would be able to write as defenders and we assume that when units didn't have the vendors mark and this was actually about furthermore above that we were suddenly obligated crack this was actually pretty arrogant ch
sometimes upstream just doesn't have the data so the day offer returned vendor was mostly sourced from other modules returns receiving module returns receiving is a fancy warehouse statistics term for the folks who was log Unsworth you boxes returned the
do you know the folks are receiving often just didn't know where the vendor was made have returned unit or office like let's visualize the late late you're warehouse worker things are coming in off the truck is a big pile I I'm and you can hope the packages are going to be nice packages that people actually put return labels on properly names time that's true right like most people are not jerks some people are jerks I sometimes people don't label things you need to figure out the mergers without that reference point I sometimes the box has a broken nothing and sometimes there's like all these permutation of ways that things actually get kind a weird out there where people are actually doing their jobs I and we're talking about warehouse workers here on the job it is not too late you this in-depth research on exactly what brand of correct just got returned and their job is to lock everything they can figure out quickly and move on to the next thing they are measured pretty aggressively on how fast they can do this and because B this is blue-collar workers in America where the unemployment rate is not so great there is a super huge correlation between their job security and how fast they can do this so if we enforce this process on them were suddenly they need to think really hard and slow down we are the ones being the jerks on also like if you a break in a box that doesn't have a vendor sometimes your business prophecies desire for data is about as realistic as might annual desire for a pony the and anyway I don't know any of this time all my team was really aware of was the light gray locally the RGB much and because all we are aware of all we're paying attention to I and was pleased very local means we ran some migrations that might screw things up blood maybe a lot I Introduction now we could end should have not done that and we could and should have stuck some new validations module model rather than running this intense destructive set of migration that unmarked a lot of things for a return and the I'm and we did not do this because when we just add these validation on our own CI failed we assumed that see I still year was US uncovering it yet another new but in this process again we're kind arrogant am instead of assuming as we should have that CI failing was a sign that we had just broken things our the real problem here is that we is isn't even that we were arrogant and ignoring what's see I was telling us is that in the process of ignoring with the I was telling us we went off and the new thing instead of talking to returns receiver I all the story that is told you about like how things actually worked in the warehouse we discovered and across retro a few weeks later and if we had slowed down I'm an late walked across the office and said you know john what's up with this I then we would never have been tempted to run this destructive migration that created the production I'm the circular putting this is that all we
need to do talk was talk to each other or
as the returns the we may have quoted when you're doing that across Sumatra later I individuals and interactions over processes and tools but this is true as far as it goes
but just is and remains the most evil word in software-development whenever it pops up you can be sure that lurking underneath someone is radically discounting the effort something takes or even worse pretending to do so as a weapon to get what they want here's another quote
and the condensed I the full quote that Camille 48 edges to form a secure and the runway and is the amount of overhead that goes into managing coordination of people cannot be overstated it is so great but the industry invented it makers services because we'd rather invest engineering headcount and dollars into software orchestration enforced disparate engineering teams to work together and this trails come there is like some dogma joke place I could take this about micro-services am I right I but it's let's get past Saturn on let's listen what she's saying underneath here I Bush is saying a lot of ways is that if we aligned the crosslinking occasion problem into just talking to each other near setting ourselves up for failure the financial is not the processes and tools have no value is that focusing on individuals interactions is likely to produce more valuable than declaring a specific set of process singles be cracked similarly observing the way the software actually behaves is always going the more accurate than reading a static what is going on there the anyway this from a combination of these 2 facts we build our mechanisms for making sure communication happens I mention before that 19 added this validation see I started screaming but and if we don't get blamed on filling some of them would have popped up with recent comments from folks are trans receiving we could have gone yellow John what's up I decided didn't realize on we decide the heroes but let's go a little bit deeper I I pardon or fixing interactions so we don't need heavyweight process isn't just building in these lightweight use like allegations it is in being able to listen to the slightly what is the this culture artists organization a culture of pretty isolated means where everyone had their goals they needed to pursue really aggressively a certain person in upper management was breathing down our necks all and that in a situation in which we simply could not be receptive to folks whether Keynes initiating these kinds of conversations and we build this Axtell cultural norm that your own work is the most important thing we are also saying communicating crustiness waste of time is never waste time so we like created this rubric for solving each individual special snowflake communication problem we encounter will be a lot of mentioned except let's move on to not using bone and hadn't patterns the now this photo
of courses from might of a favorite Star Trek episodes little known 1983 Christmas special the and anyway people forgetting are just not using established dietary patterns is 1 of the biggest opportunities we have to trap ourselves into more alive and when we're talking about and agree this is because it's an obvious example of human error but when we actually want to limit the impact that human error has our systems and moralizing leaders in the exact wrong direction and we need to look at the ways the system actually encourages us human error and building wasted lemma noted then stressing recovery not Prevention Prevention is let's add more steps of horrible waterfall paralysis recovery is let's acknowledge things move on the the you're on Candid
3 of the slide you merely intended to note that 0 boy there's lot I'm this is like a symbolic representation of the way a lot of old Orelle's projects I might acquire validation sections of the you know maybe 5 times that that I or call that sections there maybe 35 times that the and and all this is a way to like tried to build in more modeling to a State B is that integrity issues that pop up in all rules be I could say that all the model that don't have like that enormous lists have a bunch of male sex running around that that would be a total lie because we all know that those mom was also have no checks running around all over the place how we did a browsing jokes about but
let's not that leads us to blame culture and been said let's look at how and why the situation is actually terrible I'm missing the college developers I'm the and encourages developers to get past analysis by skipping thing I I I have spent a lot of two-day streaks going Hoche held what magic incantation will factory grilled work this time am I literally wrote a fixture concatenate concatenation gem to get past this problem because it was so bad at 1 particular place I was working on of sometimes getting validations is a totally rational choice which is scary but true the I'm and we need to get past this not by blaming people for having skipped validations but by reworking the systems to not encourage them to do it anymore I'm and they'll stick remember this is just as paralytic users but if we think our poor warehouse workers are later on a mention 1 it's like it maybe people are pain to work on them I'm and higher conversion rate might perhaps as suffer a little a lot the I require lot of other things up front require encourages people to either give up or just enter some fake data this actually happened very recently and I work at 4 making software using think people wouldn't just enter the data sorry grant over half of the people just do a lot of things to get past that annoying red validation box and don't want to give them that chance you don't want to leave them to that crap and we
mix called steals the full complexity of our systems hello hello home halfway through already at once you making our system unusable for both users and developers I'm
so how do we get this bit more real world usable let's start thinking of the actual business processes involved about figuring out what's needed 1 the here we are doing everything at once we using a conditional validation validate its on state on allegations of shouldn't firing given the fact that models trained and we can also use Rails custom validation context is something similar on here were specifying what context you should use when we say on we can even like a little further away from like the designated rails happy path and using that model objects and the trade off we're running through here on this
flexibility vs. cognitive load because when we go further from Israel's happy path we also increasing the amount of cognitive load we need safely that stuff when you go in and
we've just sling everything model it is super simple to save model on custom validation context mean remembering more things I Service objects mean remembering even more things than this seems subtle and you think that people just remember I look in the correct sir subject directory US they will not the other remember the norms and for real civil looking at models and they will screw up will using your pretty see service subject sign they will do this for entirely normal programmer reasons like this is a real set things right that models of and expanded nonreal Cesar subject designs but they do you increase the cognitive load that case to work with the system on and we need to be real ourselves about that and actually address the dad and had the Bucs results and you need to build a culture of caring and thorough could review etc. to backfill this discovery Pelley problem another thing we can lose maneuvering for else false is weighted rails defaults invisibly helped us use status transactions directly the identities transactions for those us were less familiar way of grouping database queries 1 you 3 4 I'm all these will stand or fall all at once on it's say pretty 3 fails great for as a fire and the entire thing goes away as thank reals called this this for free and I was call that next person I'm but reals called gives us for free than the bottom example does the I'm this is super dangerous of units A. fails not sorry if you 305 doesn't say we're going to have the some internally inconsistent at a certainly marked a ship that not all that useful thing the we can fix this by wrapping things the transaction by remembering to do that is much harder than you'd expect a arm and so we need to make sure were also heading in this class that wraps it in for you I'm we need to just do this instead going self-discipline self-discipline all the time because no 1 of self-discipline and they have a flow but anyway like the reason rails callback services for free is because I don't want to bother with this when I'm working on a feature I want to work on the feature that is what computers are for time the these things also get a bit rougher when third-party services can evolve I'm or when asynchronous traps come in and really surprise services in single jobs in many ways the same thing on a younger is going to go a bit more into this probably in her eventual consistency talk right after this 1 the I the In each case their code that we cannot easily recover from failure with with transactions I work right you up in a transaction fails that worker cannot reach magically back in time and this already completed transaction go nope sorry I'm and similarly if I make an external service call and I could comes across an error later on in the transaction tough luck are ways distributed transaction systems I advise you not to do this it is very hard I'm so if you like I don't take this advice I'm prior still want to implement the suggestions that catching mistakes they're about to give you but brief digression on this is done entirely talk therefore like contracture required to talk of the CAP theorem in the CAP theorem would I'm also is Brewer's conjecture until Nancy Lynch proved in 2002 talks about what can and can't happen with in systems involving more than 1 year uncap stands for consistency availability and partition tolerance consistency is are updates applied in the order they received you'll also hear this prop your purchase linearizability I available means exactly you think that's I'm and purchasing tolerances sounds fancy but it really is means of the systems behavior is predictable of the server crashes are a network connection cuts out you know like computers to the CAP theorem says that you can get to but you can have all 3 and Philip the CAP theorem says underneath is consistency availability pick 1 the I go could really great blog post you cannot sacrifice partition tolerance computers they everyone if the server has down then you can either pick consistency by having a refuse connections or pick availability by accepting potentially inconsistent data you not get like any other magic the adoption of time as my coworker McKillop but said all systems exist on a continuum between safety and likeness during the intercepting safety will reduce the lightness of the system and similarly you might want to prioritize lightness sometimes need to be able to accept it your dad is not the perfect if you and this is in many ways conceptually parallels the idea that sometimes we can abandon conflate complicated validation Corelex systems to make our code base more livable but this does come with the additional we let's think of the sodomy will make mistakes off and this is OK as long as you're building systems that offset these negative consequences the under this is offsets I have a lot of success running lightly order processes like every 5 minutes every 90 minutes maybe for something complicated every week reading quarter on we should do some basic consistency checks like maybe the sum of all the entries in your counting a database and make sure that you have everything adds up correctly or maybe they make sure that every water that is marked a shift actions associated shipments i and then you wanna escalate issues that crop up to real humans if discrepancies the I'm discrepancies will be found I building view you as we turned on the 1st I have worked for a company I'm not winning and where we turn the system off because it was findin I a lot of discrepancies on your point he worried that the system is the thing is a body and not your like data your data is the thing that by the end so sorry I'm work still still get better I promise slowly and another thing you can do all the things that anyway that's kind of spiritual similar stop new checking who later error tracker be this audit system this sounds like a joke but is not yet and the reason this is not a joke is because the reason we do we do on all tracks right is that we have data integrity bottoms we go bugs me and says the unexpected method whatever for no class and we go 0 mentioned about session I should have been yeah tree just a little slam the no check thrilled to deal with the proximate issue instead of investigating the reports on and ingesting maybe we should stop doing this on or even if we do need to shut parts snapped up we start walking exactly how many times prepare piece of thing but there is no us so that we can determine how severe the but it's and prioritize doing something about it but ignore system feedback the reason the systems work is that the fundamentally pouring in DevOps practice namely monitoring real-world performance and just systems accordingly I find that integrity on the other thing that's really power what the systems again were really involving humans if escalating issues of support personnel and support personnel can deal with these are too small 1 off that issues of in a way that's much more efficient than having a programmer at traced through and figure out exactly where PostgreSQL but wrong on the 5th formant of the months of again in order to get that integrity efficiently you need to recognize that software living system composed of people that I don't have situations where you can't you're down Ch values use moment that integrity patterns like validations like transactions the next step is bizarre computer nonsense this 1 goes by real fast the then seriously like the things I talked about so far in this talk on having a strong product understanding and making sure you have good inter team crossing the imitation patterns building active practices around that integrity checking and making sure get humans involved on when you have data integrity issues at the 2nd stage you when things go view the years because fundamentally when there's an issue whether it's a simple issue of light you build out because the worker failed and or heard because I don't even know like computers can be weird I inferring part is knowing that things are wrong and then fixing the thing that is wrong as quickly as possible preferably before the cost for notices or the year of FEC notices in the case of the at all of fund-raising companies to work for and you can fix things if you know they're wrong so it is
not so that the but
he did where's your product is and how these needs are likely to evolve but that's called being a business savvy application about you need to be aware communication patterns on your team and between other teams but that's called being of collaborative you need to be aware of how cognitive load and the things you don't know can lead to make well-intentioned mistakes which is called being cautious engineer and finally you need to be aware and flexible and computers inevitably computer anyways I don't think this 1 other than my paycheck is real nice this is also the to cautious that like when it that way it seemed easy but it is not there is never such a thing as easy there's never such a thing it's just a bit over arching theme of this talk is
that this stuff is hard and you're going to mess up is a complicated problems that require attention in detail and that is hard to maintain over a long time of working fast but if you build
in mechanisms that let you recover from issues quickly these issues might as well not have ever happened you don't need to fall into analysis paralysis you don't need to fall in 2 years 3 million and 5 an extra 2 waterfall steps on which will actually counterproductive and send everything the place anyway I hope you don't eat resort to unhealthy finger pointing and you don't need to do all of these struggling doomed attempts to prevent the issue next time you can just fix stuff and move on seriously
I'm betting able and I go by that's involvement where there is very little programming there's a lot of cats and also some feminism In this talk slides and a rough transcript we posted at the the cable dot com slash talks at some point with the next week or so of the I work in a company called was divided eastern San Francisco and we're not currently hiring but will be reopening hiring for senior developers soon and if you want to help mentored some of the best union level developers that you could be published work with and I highly encourage you find the hum I organize a group called moving DC which is a casual office hours type thing for new where programmers in the DC area and if you'd like to do something similar in your town talking about it's not hard you just need to show of and I got a few minutes for
questions on on the questions of them are there any tools that can help us check the data in our bases I and this is going to sound like a silly answer but sigh late seriously whatever asynchronous were scheduled job rhinorrhea already using anyway I you build these all systems I'm talking about 5 8 just having a thing that e-mails you works and the about and if it finds a problem on every 5 minutes it'll be fine are more it's really expensive calculation may be less than that I don't over think that the important part is that you build something quick and sustainable common you start incorporating even start incorporating more things over time I and so the question was we have these transactions script objects that are external the rails to help us remember directing the transactions we run the risk of forgetting use that object on which is very real me and there isn't much that can help with that other than a strong culture of code review like I've seen some people make more complex things as a popover escape years ago really can't about like monkey catching rails to the area great you if you step past things it gets more complicated and you wanted to arm but even having laid this cultural norm of every single object in this folder must inherit from this object army does do a lot to help because then you've made this obvious point 3 things will jump out at you if they're wrong so people are more likely to catch here I am and so the question was how a lot of the times when we're trying to deal with particularly bad queries that I'm Active Record may generate by default we wind-up of resorting to Ross equal on which of course naturally bypasses rails is on callback mechanisms validation mechanisms etc. and therefore also bypasses the business rules these maintaining arm and so how how you deal with this and but again there's no light perfect answer here unfortunately on 1 thing you can consider doing I it is and Israel's can't so I'm going to like get in trouble for recommended not be database agnostic but don't be database agnostic folks I'm like it is OK to use some post-crisis built-in check mechanisms for example push more of these things in the database on this is going to lead to some flexibility issues like you can start running like check columns only work within a table of anything outside a table no need to start thinking triggers and having worked on a system that actually did need to use this to maintain data integrity because we were pushing so many things in the sequel for scale on you can check out my former colleague talked for a little more about the other scaling issues that we face plus DA I'm it's like today I the in you really careful like Rails doesn't have great feedback mechanisms for her arms reporting back the results of these kind of triggers scaling for example back into rails in a way that will make this easy the bonds on me and so this is a really hard core choice you should make sure you're making only if you really need an ominous a lot of time I would definitely advise that you think really hard about whether you are best served by on doing everything in sequel or extracting things to service subject imposes these business rules or are at I'm whether you really do need to push things and that is sometimes you do but it makes lot harder to up I sounds like that the thing that I'm out of time and thank you all to thank there're if my
rence yeah