Logo TIB AV-Portal Logo TIB AV-Portal

Mobile Canvassing with PostgreSQL

Video in TIB AV-Portal: Mobile Canvassing with PostgreSQL

Formal Metadata

Mobile Canvassing with PostgreSQL
Alternative Title
Mobile Canvassing and PostgreSQL
Title of Series
Number of Parts
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Production Place
Ottawa, Canada

Content Metadata

Subject Area
At Moonshadow Mobile, we visualize geospatial data in a custom geospatial engine and analysis framework inside a browser. This is coupled with Ground Game, a mobile canvassing solution that lets managers and canvassers easily mount a campaign. At the heart of this solution we need a featureful database with a high level of data integrity to quickly store and retrieve customer data, all of which PostgreSQL provides. Our Ground Game mobile canvassing solution is available for iDevices, Android devices, and any device with a good mobile web browser.Campaign managers assign walking lists and questionnaires to canvassers in the field from our web application, who receive the data on their mobile devices and fill out questionnaires while going door-to-door. Data can be collected even if no connection is available and the data is synced as soon as a wifi or mobile data is available. On a typical day hundreds of canvassers around the country upload thousands of questionnaires. Our base data consists of the voter files that are collected from states and counties, augmented with psychographic information from commercial sources. At any time voter files,phone numbers,addresses,and emails are updated. Clients augment our data with their own information about donations and membership information, along with mobile canvassing. All these data streams need to be processed without causing any downtime for the campaign managers and the canvassers in the field.PostgreSQL plays a very important role in our technology stack. All canvassing data,walking lists,canvassing users,assignments,and questionnaires are stored in PostgreSQL. We also use PostGIS for the creation of the data and the assignment of voters to district zones.In this talk, I will give an overview of our architecture, and focus on how PostgreSQL has helped us deliver our product. These reasons include custom features of PostgreSQL such as bytea,json,and custom types, addon modules such as crypto and ltree,and custom functions in languages such as plperlu,along with WAL replication,hot backups,all of which give us security,peace of mind,and the ability to more rapidly develop features. Finally, we look to the future and eagerly await integrating new and additional PostgreSQL features into our workflow.
point time cellular completion data analysis Average completion programs Computing connections computational production powerful voting Location message-based voting Tower shadow input circle sort write
Cover track states Gender time sources analysis analysis Dogs voting daylight saving time platforms platforms
states time formating shape clients part programme static different systems hard Arm width formating Development files binaries tournament storage traction processes different sort web pages server statistics files analysis storage browser bin hypotheses number powerful Representation level continuous report platforms Authentication validation Tiled analysis catalog Transformers applications shape software platforms
polygons Context Gender states code time formating sets water shape clients dictionaries total area redundancy geometric different Display record processes Multiple area algorithm width formating files point feedback traction functions cotangent sequence Types means buffer cycle sort record geometric point Runden table files link Sequel encoding ups analysis coordination fields powerful training number Central level addresses form data types data processing focus Multiple matchings validation Gender polygons analysis experts fields print incident parity shape CAN-bus words voting mix key Display table spectral
reading presentation mapping table files attributes orders topology Query sets Recursive curves presentation app key mapping demo formating Development analytics ease of use applications Regular Expressions Query Web Development topology interfaces cloud key sort table
track control formating storage part sun mechanisms sets spaces Sum area unit mapping real interfaces polygons storage ease of use Types Indexable topology orders Universal interfaces sort table simulation spaces
hard choice polygons randomization Actions states time sources formating Relational ones survey sets part total sun programme Dogs different analog Query sets file system Universal source code Bas systems area joystick hard track Venn diagrams formating load files storage functions measures structural CPUs static interfaces organization structure record track randomness server app Super services addition files Development patched storage Coloured fields crashes Central selection surjections user interfaces app addition Multiple scale interfaces survey neighborhoods polygons states ease of use Databases total lines applications timestamps words Universal platforms table
polygons email server time The list analysis storage rules van voting Display Universal circle addresses data types app Arm interfaces survey neighborhoods states The list analysis Databases applications connections mining Computer animation software circle Query Universal Synchronous Right sort WAN addresses
Actions code Gleichheitszeichen survey sets TSP part Bugs web Dogs Types Synchronous different hash circle model comparison Security systems script Development moment traction bits functions Types management processes hash testing sort Security Results server backup addition Development recovery password drop number production Doppler effect testing Cats sort backup platforms Authentication app comparison Multiple interfaces analysis high availability Databases lines case functions password Data Centre platforms cloud
the next Villarreal 1 of my name is we see powers I I work at this little circle in China mobile and we do mobile canvassing and data analytics and geospatial things and hear talk to you about how many of these posts rescue all with canvassing all of so just the basics of canvassing canvassing is all about targeting population you want to narrow down to like your supporters were people were actually interested in the product if not doing something political on and you know and sometimes people do it for community outreach and things like that so basically 1 there at a population of people that want to and and might be interested in your message most of the time it's used for things like get out the vote and everything but the main point of it is the targeting and you also need to guarantee your completeness you need to make sure that you've you've tried to contact as many of those targeted people as you possibly can so lots of us walk around with this really sophisticated tiny little computer our pocket on and it's got all sorts of radio connections and it's also the most of them have GPS chips now and c have radio connections with Wi-Fi and with the mobile data and just with the cell tower and things like Android and IOS use that pinpoint pinpoint you and everything so you can you can know someone's location very closely but it in an area and see how they move and everything but you can also write custom programs for them up and the big point of this is you can really leverage this technology for lots of Austin efficient canvassing so to be able to
reach the appropriate publishing you do really good analysis and they need to have a really detailed analysis and on people mix together political daylight your voting history your Registered Party how many times you vote of of donation history things like that and demographic data like how old are you what your race what's your gender or things like that psychographic data which you can get from like commercial sources and things like that works that you mention things like OK well you subscribed outdoor magazines research greater religious magazines or personal or or like pet things and whatever
I'm sorry on and then people have custom data that they want put in because sometimes even though we have a really rich data source the stuff that people don't wanna cover like there were some people in a state that wanted to die to target people who own tracks handguns ends I love dogs and really useful data platforms need to allow for stuff like that all right so this is kind
of an overview of our platform everything on and there's kind of 2 data parts to it on there is this we have this network file-server on and that holds up what we all refer to as as static data it's we take large amounts of data and we compress it's very very very small and this binary format and it goes into this thing we call query server which is actually our lead programmers I think it was his PhD thesis actually I'm not sure if we ever finished his PhD thesis but yeah they have that before came along and then and then there's PostgreSQL which holds dynamic data which is data that's generated by Kabal doing analysis on the course here and I you know it's saved for future reference and everything it also holds like like customer data like names and what applications they're allowed and everything like that and it talks to the authentication server on so there's the analysis part which is basically this connects to their work and then there's a web client that runs an all you examples of this arm and then there's also the API which also uses a bunch of other other sorts of technologies like we have this so where that because people really really want to be able to do of from canvassing and they just want to be a local something up on a page and then there's the mobile device stuff that I talked about earlier and you know just a way for people to export out stuff on in various formats if they want to which is the reports of part of so this is kind of what the analysis platform looks like this is a one of this is like the biggest actually on the end of the course regenerate these tiles but also generates the statistics over here on but this like the sidebar how this is laid out here a store and breast whether or not these things are enable also stored there on how this is displayed white matter all sorts of different things and this is just basically a representation of all the population in the United States because most of our customers
use that on but for the most part customers you're interested in honor of canvassing on a much smaller level most of the work within states and this is this is Oregon and and here's an example of a quick little thing then I threw together for this talk of a survey of whether or not like catalogs and everything and I'll go over in more detail what this means and why it's important and everything by has PostgreSQL with all of this up and put stress is actually really really really powerful platform for cleaning and transforming data on because data clients have all sorts of ideas of what's what's cool and what isn't and what's OK and the system isn't necessarily OK Mars because while the creatures kind of cranky of and also gives us durable dependable flexible data storage like we can use things like biting hinges on all sorts of other things and we also use it to you on offload features from developers because they're really busy but not that I'm not really busy and everything but also yeah everybody's really busy so if we can offload things to post processing you can centralized various that's that's always a win especially in a little star so customers have been a lot of different formats of the most common are these with CSP in shape files on fixed with files are basically of and you have a wine of continuous texting you need to know what the offsets are and of the number of characters for each 1 on CSV is basically any sort of delimited file and then shapefiles of this particular geospatial format and it's got like an XML file in binary and much of the stuff but the common thing that all these have is most of the time the really dirty but they're really like people of format things we're the have columns off or they just won't include something or whatever and it off often requires tons of cleansing nipulating validation all which is chris actually does really well
on there's also this large ecosystem of tools that are totally awesome that we used in different places of PG later but is really good for fixed with files you can just make a definition of a definition and a table schema and then just throw it out at this is really really handy for things like the census data in the United States Census data of a lot of it is in fixed with format and of its it's miserable the deal with the peeler makes it really nice on we leverage a lot of host yes because we use a lot of shape files and she failed find areas areas are ways that people use to target Nureddin populations so knowing areas whatever they may be are very important on and we also use the Copy command because of just being something into a table and post-crisis powerful in and of itself because you can have posters validate 0 yeah this is this column actually is an were it doesn't have any word word it has to be the in it that doesn't necessarily the right type or the type that they said it would be on word slightly malformed also so you can eliminate things like that but if there's a lot of really melphalan data you can use PG loaded a kick out on now malformed the rows of the other end of the address level most of the time with the geocoding yes but for that alone we need to actually ask of what you have to ask the client what they wanted and so you have to you have to do it do a feedback cycle which is where fight we back things all you know we have have that that role bad data we send it back to the client go what you want to do with this because of the clients of the client own their own data we just provide a way for them to do so important do analysis for it but yeah we make sure that only they can see it and everything but it's it's effectively there data because I I don't I don't have enough time and the data so there are about 1 year we generally destroy the data when and when a when a relationship was terminated including all back ups and all of this so of variant voter all all of those words neither 1 actually they're they're about the same they're they're they're all they've all got their own works in different ways so all this is a really cut down example of something that I We actually ran into with the client and the data so I can go along a bloodless and oppose stress and everything and then I do on a group by and I start finding things like this and so you know like OK gender male female G book what that is that guy's spectral or does that mean you know it's like well OK I think if this is free form I could kind of see this but like 1 and like so so this thing so we were able to load the data and this was probably about 19 million rows show and there was 1 per cent of them that have this these particular sorts of problems and everything we're able to isolate them export them and get them back to the client so they can actually do something with them and and in the end we ended up just bundling means so they came up with unknown on which if we didn't have any claim guidance I would have probably suggested anyway because it's fairly nonsensical of but yes because resources validate all these assumptions that customers have about the data of a new customers will provide a data dictionary and they'll say that a column isn't that isn't always necessarily true on and being able to push back things the invalid also helps customers improve the data process and really really good data having excellent data is absolutely key for doing good analysis which is good targeting which will make canvassing that this is all very important thing again shapefiles they also you can import the Wishart T-SQL it's really simple it's like
shaped tedious to all minus that's 42 69 and then pointed to shape he gets sequel you loaded and you have a table and then you can start doing everything that has recently got out with it on you can also use POS GIS for validating cleansing shapes and normalizing them off because sometimes I shape files have slightly invalid geometries in areas and everything you can clean all of that up with you know the stack of functions and do that on it even deals nicely with encoding issues which is very nice that's very rare so this is kind of and this is an example of the of the county shapefile for Oregon from the US Census and US Census has really good data but even though they have really good data they have really inscrutable names for things also like this is basically a central the central point of Wayne County all of which were in front of it and then you have this thing like the GEO ID which consists of this little number 41 is the state that's code for Oregon and 39 is the Fips code from Wayne County on them and a lot of our customers really don't care about little details like that they really just wanna see like OK here's an aerial intended use in it on lots of them really aren't that interesting like they don't care about area lander water all that stuff and all of this basically needs to be shown to users in a format that they really understand on and so we make use of all of these different of of post yes functions and these are the top 1 of st is valid is is the geometry record a nite on why is it not broken with St is valid reason on st make valid is use this uses an interesting set of algorithms to try and force it into a well-formed geometry on sometimes that comes out in a multi a geometry collection format which from our engine doesn't understand so we use ST Multi to clean it up because it will force it into on multi geometry on sometimes people want to like OK well this is kind of bumpy and kind of weird and and someone handed this economy expanded ever so slightly to make sure that we get everything and you can do that with STD buffer and you can control that on for exporting and making this pretty we use of STS text which is the incident in this well-known text format which is basically just longitude latitude and everything on sometimes they're like use St centroid because we we try and print everything pretty and all of that and we try likes under the name of it on the display over the centroid and sometimes centroids are actually in the middle of something so you have to check to see if it contains and if it does then we find out of all points inside a polygon and then get its x and y coordinates and use that on we also do intersections that sometimes people have polygons and they'll be like well you know these really should have been together so on the kth in use St intersect warmer together and then export that of a shape found use these to make things that we call on match all but we normalize fields we normalize the training data because sometimes it is isn't always in a pretty formats sometimes it is a likely in County sometimes it's just like of 39 and then we have to go make a lookup table and then instead of 39 prattling counting on this this stuff like this was especially bad wind of all of the states were going through the redistricting and they would just published shapefiles with basically a sequence and then number saying like OK this is this is district 1 2 3 4 5 6 7 and then geometry and you had to know from you yet another context of all of it from where you downloaded it and what it was and you know you needed to know that you needed to put down congressional district 1 2 3 4 and all of that on what we do this so people don't have to be GIS experts so that they can just do their own thing on and that lets them you know focus on what they what they really want to do which is isolated splay people in areas important to them and then narrow them down for targeting and so this is an example of a match there's like link having here and you can see where everything's all sounded like there's a party County in Lake County in Klamath and everything up and we have all sorts of different other different areas here so that you can choose from and you can do all sorts of mixing and matching of the data on so the whole layout
and presentation of this thing is very important here all of it
really needs to be easy to customize and everything because some people in some features some people need others some people legally aren't allowed to have some features of the analyte show some data and some people need to see this data and you need to keep everything straight on so we need something that was very easy to customize and working on originally when I came along these were defined in flat files with this on Jason all and or web developers did that because they do know JSN they had Borge a song on because it's really really easy to work with Jason on so I took it and did some manipulating with it and then decided on using L tree I to take Jason and stored in tables on it makes ad-hoc queries really really really simple and we started out at 8 3 moved to 8 4 all I showed some of the demos recursive reasoning went kind of light so I was like OK well what would do what so sort of pre format that you would actually wanted to deal with and so I went through a couple of things and we can we looked at all tree and we use that on the it makes like it works really well actually 0 it's totally fast it's also we again trees with 22 thousand entities out in the cloud about our somewhere anywhere between a 100 in 100 ms to like really fast if it's in like you know 16 ms or not MS it yeah yeah 100 or 60 ms sorry all of but gathers makes ad-hoc queries really simple it's very quick to read it's very quick to pull that out of and it's quick it's easy to update and place on and it's also easy to append things to it if you want to all right so this is 1 of those things that we do we we call like how and application laid out application attributes of the and this is for labels and was Oregon I chose the smaller path so what actually show up on the slide next on but yet a lot of developers just wanted to deal with the associated your regular expression this thing on here on the path and isolate stuff on and you can just like if we need to change like of being maps license or something of the license the API key it's just a single update like that and it's actually it actually works really nice and a lot of
ways and this is kind of what the table looks like and here's the the map part so like just forgot dealing with the map there's the adapter which is 2 which is being maps you can switch this and out if you want to with an update on this is basically centers the map on top of Oregon and puts it in a particular type because the map can have like auto or whatever you think you can have very very fine-grained control what's there and if you enable features like polygons in sampling and universes and things like that and so on and you can preserve like the tree order here of J which is actually a really nice on so you can you make
sure things are nested like here's counties and underneath counties of nested underneath are just the like County commissioner districts and everything and so on you can people can click through a narrow down to the the small areas that they're actually interested in very easily and we were trying to keep this as simple as possible but there are downsides everything because it's a lot of work to cross it into 4 right yeah departs the JASigning and then you can turn it into a new turning into a road and there's there's a triggered update the revealed tree path and you do other things and keep track of stuff there's there's a lot of mechanisms for that and because of that it takes of extra storage space because there's extra indexes and triggers all sorts of things like that but all in all we what on now once we have the the data once we have the data and we clean the data in PostgreSQL and we've defined without House of how the interface works have actually had people actually use this what apparently accomplishing real work with it so here's kind of an
example that I went through of this I took our polygon tool and I decided I didn't I was interested in any area that we had predefined so I in my own this neighborhood in Eugene on time and so I just I just clicked and draw it out and people do this all the time in our system but 1 user has some patiently drawn at about 400 years all and so you can you narrative selection of this to you know this particular neighborhood on and everything and it's really hard to predict what people want to do like when we originally envisioned polygons we never expected that someone would sit there patiently dry like 400 of them all so flexible data formats like J on a really really really helpful because we have no clue what people are going to do with something and we just can't predict enough of that stuff and we use J songs in a few places but but 1 of the 1 of the really really good ones is for random user state it makes it easy for them because they're used to doing which on everything and all they have to do is updated field and everything on and it's easy for me because I am not always making database patches and running all the tables a table and column table and column of so yes relational and
semi structured is really amazing it's like peanut butter and chocolate and so all of it originally when we came up with this table and this is just part of it because the the values with J. Sonner really found of the really long and the really big on and off on the side here off on the left you know you'll have you'll have the relational things that you need like the user name and the at name and the customer name everything that we use to keep isolated we have like time time stamps and what data this applies to you and some of some other things but we have all of this relational stuff that keeps track of all this and then we store the stuff that we have no clue what they're going to do with In J. Sun like and here is mine my dog survey polygons and it's got the name and then it's got the coordinates here and everything I'm also I had on a couple of measures that on and some of my color buyers and stuff like that so like this all of this shows like OK I had this thing said at 1 time so of and I made custom colors for things so it'll keep track of all of that and we have no idea what selectors people are going to use and what colors and 1 everything and but people definitely wanna save all this stuff and they wanna come back to their own analytic state when you know that the go the come back they wanna not lose the stuff so we store like this and it was really really well on I at this worked so this work pretty well on in for we we did this actually just with text fields just because it was so handy on line you know which is having it being taxed but there's problems of invalidated if there's a bottoming out there right invalid Afonso which sometimes causes problems like when you right when you write analogies on and then the other comes back and it tries to read the analogy assignment there's times it will crash and that is really sad and so that's 1 of the problems we have because it's just text also trying to manipulate anything with anything inside that g inside the database was really really obnoxious like I tried writing on POV 8 functions and of appeal Pearl functions and it just turns really Greece on by 1 of another way that people also save the state is they make this thing called the universe so I took that solely on the selection and then I saved it in what we call universe was really just a grouping of people on and so guess like is said of the universe is a set of people on it's based on that static data that stuff and it's created in the geospatial engine we have on but saving post rests on the and it's done that way for some really important reasons that I will get to but this is kind of what it looks like so you have you have that non non that the relational data here on that keeps track of everything name your app naming your data source customer and all this other stuff and then have a whole just a whole ton of lights on in different different ways and places and everything and that's basically just frozen application state on so like when it saves that on and those those pre-service servers go up and down the dynamically scale with the load so you're not guaranteed the even to be on the same system that you were on like you know 30 minutes ago because the application 1 on and then it'll just sort this state back up and you can start to start working on lets keep track of lots of data in a really small place of it actually comes out due to like the compression and stuff to a high 109 bytes total for 600 people just making it really really really small also that's by and then you have just having the additional row data to describe all of this 1 and we used university define assignments canvassing on in a in the words 1 system that we have for this before I came along these receive directly on the file system and I had just the tail end of experience and this and that was kind enough for me there were lots of problems especially if the app crash and sometimes it would crash in the middle of writing were doing something and that would corrupt the little binary file so that these universes were saved and so on and we had a crime that back them up like every 10 minutes of because that was basically how it was handled before I think long on and we have to restore them like because customers to call up and the way I can load by Apple on all my all my walking listen disappeared you know terrible terrible things like that so yeah restoring them sort on and we also have these really opaque names because of what it was really easy for the programmer just write out that way on but post recentralize all of this because we have the service they go up and down and up and down we can't even get we can guarantee that the those files still be there if they renowned file system and writing to and if that's it will take for ever on so PostgreSQL centralizes all in 1 place keeps all the data together we don't have to worry about corrupted files or anything like that because it either makes it is pos rest or doesn't if it crashes someone has to redo the thing but that's way better than than having to wait of 30 to 45 minutes while I make sure that I got all the right files in and hard and everything's good on all all this really really has saved us a lot of customer time because the not waiting they're going like lazy warning a fix this and then develop a time because all the like this crash we really need to fix this and sysadmin time where I'm like heart which file to OK all so posters does this is this is really amazing on CPUs vs. lets us do this and we can get on doing other things on so we take a universe which is just a group of people and a survey which is just a group of questions and bash them together you can on you can create a survey and a user interface and it's basically just storage a song because you know you just you can order the questions and you let people you know manipulate them we order them into other things on this kind of what the interface looks like all so you can make a survey and you can choose whether or not you want multiple choice stuff you can choose you can change your reader named question all of this all of these things on and once you make that it's the word with associating data in PostgreSQL along with some meta-data about it too on and then you take and you assign it to a universe which here is referred to as a walking unless there also terminology in little clarification I think probably in the organization of them and then you can assign it to a particular user so once all that's finished you're totally ready for canvassing and you know you get the and I lesser Android app and I then log in and start downloading stuff onto the device and this is what it what it looks
like and so you can have multiple campaigns we keep all the straight from the all of this data is downloaded onto SQL like on me only of you can tell when you're in a saying you can do other things like that this is my little joke investor I and a screenshot of my phone all right so you once you get that you'd like these walking with details so this is mine this my walking list and then I'll have like questions but this list out like every address because of the University of some gets turned into a CSV and then downloaded and parsed and and its associated with that Jason because that's all past also on the I there and then put in the luster like database so that you don't really actually need a network connection while you're doing a bunch of stuff but it really helps when you know you wanna send it back to the mother ship on so this is basically just a neighborhood in Eugene arm and this happened to be the closest addressed when I had the idea to do this when I was out shopping at Safeway and so on and this is basically the interface here that you look at the time so there should be a question and then you can just pick things here on whichever is in your your multi-choice on and then you have freeform text but that's really hard and display for people on so we we we let people exporter but we don't show it on any sort of interface and of the e-mail address was also hard to show and there's there's some weird rules about them so we just don't show that it's also totally about the name of the person that was the person of the house I was in front of all of so
within that you can either choose to see things back purposefully a in the background on and when you do that it goes through the API Server than those in the query server and then it turned into another sort of frozen applications papers stored very simple but separately on in abiding just like universe and this is really really powerful because it allows this whole full circle of analysis all right
so yeah I you know park over here just pushed on some some buttons in my car and everything on my phone I'm and randomly answered some questions on any of you see like some of them like cats and dogs someone like me the I just because I randomly chose that way and I and my little joke a Mr. canvas 6 people you'd also see that I missed a few people here because they're still brain which is the UN can color on yeah I know but you get feedback here based on like your answers and everything about like what about the survey is popular or unpopular and you can use that to see who who within within a group is likely to support you isn't on you can try out different questions and see like the results of that because people do that people can use the same people over with a different set of questions just because human psychology is weird and if you phrase something slightly different of some people are more likely to go 1 way or another all but also this basically lets you knew that whole full circle of analysis that I was talking about and that's really powerful because you just get to iterate and iterate iterate and I'm trying to get the best result for whatever you're trying to accomplish with your mobile camera so put stress is also very convenient convenient as well it has all sorts of things but it has this flexible types and everything in flexible ways you can model data and everything and it's really kind of a platform unto itself actually because there's just like there's an ecosystem around there's contour this everything on there and it's of controversy especially full of goodies likely crept up on and it lets you use of you can also you impose resignation user-defined types on anything mix them with functions and things like that and 1 of the things that I used was made a password type in a couple of comparison functions on to due to the password hashing because it had been done really inefficiently in the Web that before and it was inspired by the two-pass contra model all of which was actually really cool but didn't quite do what I want on this will trigger that basically just have the password is a comes in and a password is equal to something or not equal the that the hash is equal and unequal there's really nothing else on the other we use it to like authenticate people and everything and it off with a lot of work from the developers because they the original that had a a really weird way of doing it on that was lots of lines of code and it was basically just an M D 5 and a number and that did that wasn't really all that good as far as security but even for this the security considerations that on their pointed out to me also that of the posters process can see that in all I can see the plaintext everything so far so if someone they will to get like and look at your logs or they're able to compromise the suppose reciprocity in some way they see people's passwords and do things like that and that is definitely something you would wanna consider on with doing something like that and we are totally totally screwed if someone gets on a database of so we know we protect it really really well so this actually works fairly well for us anyway on another also convenient thing actually is on everyday knows there's good for a high availability in disaster recovery everything but it involves these extra servers just kind of like waiting there and doing their thing and whatever on and that is that is always look at the management all especially like if you're in a start and the cost is an issue and like so what the servers doing quite well there there in case something is really really wrong but what do I mean is something really can't go wrong there can not have this so you have all these questions on and 1 of the things that I I do with that the slaves I take snapshots of them and I use this to local on nuclear because it's it's really really handy but it's amazing thank you on the TI so I have 3 1 standbys from R 1 production data center and 1 that's the lady so just because you know sometimes people can and can't do something weird sometimes a napkin everybody inefficient mangled data drops of whatever I I I can play right up to that moment and then stops and have a database that's last possibly a little bit of data but isn't nearly as horribly mangled as as it could have been and then we have our art staging in eastern our testing in in west coast is centered on on the cloud and this lets me keep those in sync actually really well on and I just scripting this during weekly snapshot lets us keep staging up-to-date so that when we know we were rolling out code that's OK this is this is about as close to the recent productions snapshot as we can possibly do and on we know that OK this isn't really going to do anything to terrible anything on with everything i on nuclear has a bunch of functions for this and stuff that lets you just take a snapshot of a slave on but I use scripts i've wrapped it so that people who are me can do this I because you know sometimes I wanna sit at home and have gotten much by pointing out that so they can do this stuff they they take the the back up the entire on wherever they need to even for the developer systems and off off here on and off you go but it takes about like an hour and 20 minutes from soup to nuts on the end for me at least with the start up and that the consideration of all this stuff too so much easier sell sell the say this keeps us safe from disasters but it also lets the developers find bugs that they wouldn't otherwise found on because they can have a snapshot of production and they can have their code on test and they can see the differences in go like well OK this is the data is pretty much the same here so this weird you I think this weird thing work counteroffer this is probably because of a code but not because of something we savor or because the data itself is off you can do this lovely AB side side comparison and it's really handy on it also means that my developers are testing a backup on which you know offloading some work under them is always nice plus it makes them happy I'm so post rescue and its ecosystem of tools are this really key part of our infrastructure we use it for a lot of different things we use it to keep things on we keep things running these Doppler really hard work because it does things like durability really well on and as a bunch of other stuff for us we really wouldn't have is nice system without it so if anybody has questions about that I guess you the traveling salesman problem and they want to also generally if people have a that they want to follow so we have an interface that lets them dry out in whatever way they want because even if we algorithmically try planet which is kind of an interesting and so cynical problems but also very hard problem of were not to make people happy enough for them to 1 uses so on we just we let them drawn out because that's what they want exactly exactly so right yes so we don't we don't wanna do that we we we tried it out in the eye and ended up like that people don't like it yet all of you know that's basically there's there's phone numbers and stuff and the data were in the customer data and we just we just display it in a slightly different interface on and so on and yet we basically it basically let people do what they can do on the on the mobile device with multiple phone numbers in a way that is really all that for and they find that really handy actually on the case of the any other questions about stuff well thank you thank you come and if you wanna get in and get a look at any of the free stuff that we have you can look here on and I can give people will link for that but we have stuff that's available public just to look at and see what the systems like I think you all