From Legacy to Admin

in a in a year and a half and I move around a lot is lazy and that you guys are coming up of know you add many other options so that the end of this talk and I hope to be informative and about inspect each of which is a general comment that helps you of reverse engineer model from an existing database on

within Inspector Chris for all but I go by monodomain Chris of when I was practicing this I ran along some Fig kind of fast to trying as much information as they can on this is me there's a

professional Christian and the regular chris so that you can talk to me afterward so you know your

talk to graduate not professional cross but right now

it's a professional cross on so something to the talk that I wanted to give our don't right so don't write code unless you have 2 and my name is chris not data come up and if you do something to do something hard which I know that you probably doing it wrong and this and is something I realize I started using Django initially on when I say so when I created a custom command to populate initial data and I realize that there were fixtures of so this happens a lot when you like when you start to work on something you don't read all the documentation or you see some cool part of it you wanna do not realize that there's other things of so my advice is you know to spend as you come the general con and go of Gaia Iousy of gold forums and there's a lot of rich information there's Fletcher such gender and that's and there's a lot of like custom

apps uh templates TL have like their suggested ways of doing things so inspected B is 1 way to save a lot of time and if you work

with legacy systems so instead the reads tables and columns not rows in a database and produces a model stop I've standard out of have to manually write this code and you could you could well you know create a class that inherits from all the model and give an explicit TV DVD it like mean and and you could go through every table in your database but this just jump start the process for you it's not perfect so we'll go over why what what we got do afterwards but

it works pretty well and when you need this I and 2 scenarios have come up new this idea that I have legacy legacy system listed in this Annex 1 admin interface to it of so foreign keys are grade of the primary key constraints a great result mean always and strengthen my sequel comes with but in reality you have like an additional of validation layer to let the system that you want keep it clean state you can you reverse-engineer model at a couple validations and you'll have this slicker admin interface of the general comes with for free on the other use case that I've come up

with is a pretty management commands us so when you create management commands for going through much of rows of a database and making sure that 1 field is less than another field or you won't find all of rows that have you know a certain value the or em comes in handy for 2 for doing these things and writing custom sequel is sometimes hard of particularly if like you know it's not not designed to be a programming language designed to specific things in sequel that's another reason why this is come handy in the past so

credit created a fictitious scenario of y of why I'm doing this demo and I have a legacy system with a legacy databases and I can't touch the code for whatever reason I and I need to be able to manipulate data but still keeping a clean state and so I can have the source and there's a

lot of reasons why you India was at the source you have you have a source of to third-party product it's written rails or of you know the source principle it's impossible understand also at the end of this demo I

want I want to be able to have an admin interface that's there was little work as possible and you will have a basic right operations and 1 be a little leave the existing database intact more or less and we'll talk about how we break that when we have to break that so many ways to ways to get around it but a more or less what that leads the system it's interacting with this code to to be the same after

this also the tree from the docks

insect avait and database and whatever's pointed so the 1st thing you'll notice here is that of whatever your settings of name is pointed to death with can introspect of and it's also in this state the statement doesn't seem like it says a lot but it actually says a lot the script once metadata asynchrony model for each table on so something that you don't think about is that you often have more tables than models because of many to many relationships and inspected the inspection don't don't understand the concept of and also the most important part of this is this

feature is meant to be a shortcut not is different among generations so I am I created a

fictitious model yeah so it's pretty

basic there's a user table I made a lot of like egregious errors don't blame me I designed this to be terrible on purpose so that we could understand what would happen if you designed the is not yet a work with is the terrible manure expecting them they're not to be perfect so every user table it has 1 primary key it's our incremented it's not null and has username and then of course it has a point X that's work of we have a company table has 1 of the primary key is Otto incremented if that in a minute self-referential so a company can have a parent company and so it's interesting corner case and then this goes on before a user can belong to many companies and many come and a company can have many users and their related by this user has company of many many table so this relationship in in uterine Django you have a user model that has a many-to-many relationship or a company or vise versa but there's really no way of of work of gender inspection to know this so we're missing at what this action produces a 2nd and another thing that's terrible but this referring to the Django at least that is that has 2 primary key so has composite primary keys which is gonna be a problem for us some of the y the of Agenda basically since the you have every single model this in the of 1 ID field that has a primary key and this is doesn't fit that pattern and then profile budgetary thought community worse has a triple a composite key and there's a role here so role is meant to be a foreign key to this table role but this is of our car and the name years so as to match the role in here there's like completely worst interests and you can imagine there can be so many problems with this obser anomalies deletion anomalies like you name it so that's but smashing model and then just for fun if we have time at the end and go over this table because I created like the worst of field names possible just to see what they would produce and so the source of inspect the

be all and this is the source of new it's good size I guess so that there's 2 primary lives the 1st loop other users connection that infection table names also connection introspection is a of connection specific introspection function to get the table names so depending on what database you're using it'll get it'll user differ command to get back at list of table names but then once it has that was the table names of its agnostic this code of the rest of goes agnostic to the type of database you have so I looked at the source code for other the different introspection definitions so my sequel oracle and posters because all of them and you look at the documentation it says my superimpose grows over pretty much was before spices and and oracle has the same function introspection functions methods defined to introspect database so I don't get a chance to test oracle but I don't see the reason why when work and I was looking online it seems like a lot people driving to work so maybe I can look up that later in the talk about that but so once you get a table name and at that 1st primary loop the secondary part of the loop of the inside that loop I guess relationships gets indexes and then it has a secondary loop to loop through all the other columns and at the end of this and ardently do the job of the screenshot but the end of this it generates a collocate yields a the output of to the use an output of the column name to standard out so that already used so this was a little quiz I made

a these are the intersection Pacific commands that my seat so that gender uses that to get table names so my he goes to show tables are select tables for Oracle and something knows about these these each of these commands is not guaranteed to come in any particular order social tables just gives you a list of the tables of it doesn't give you the table sorted by you know when they were created and use the table sorted by of the name and all these are the same so we go through that loop it's basically going get up and there's just no way to guarantee what order it but so this is

what we talked about before the we use the connection introspection about to get the table names relation of new indexes and the columns but then we have agnostic at the next layer to generate the models once we have all information from our a connection and this is the many-to-many fell of

like I said this should map with a many to many on the the use of the company field depending but this maps 3 individual models so let's take a look at

what would happen if we do this so Python managed to apply it to the show so I created this in my sequel so we'll take a look at us so these are the 6 tables we started with and we haven't think the database so we don't have any of the gender-specific of tables so the question is do we wanna singer database or do database introspection 1st the database introspection looks to the database and create a model from the DB creates ah since a database of to the correct point of that were out right now or whatever we have the and and uh right now so if we run our sink database 1st order create tables that of gender needs to be run at to log in and you can't older users but then is we run uninspected days after that those tables will also be in our database and they will be introspected as well and that's not really what we want because those are those models already defined somewhere else around them inspect B 1st the so like I said goes directly standard out of which is not very useful but you can take a look at it of things so tells you to do things which is nice so I had to be be giving you talk industry this you rearrange the models you make sure that and everyone has a primary key and if you want you can remove managed falls on the doesn't really say with that's gonna do besides its can allow that gender to have a lifecycle and allowed to rename the models but you're not allowed to read DB tables which is like the Met option to explicitly state the table name is to override the convention of and then there's this line which you know I'm sure won't be important all you have to insert the output of gender and sequel custom app name your whatever so let's go ahead and put this to so I created map for this you have and let's assume we

produce this is same thing but worse in a file now of and now we can run our safety the the so the 1st from the run into is of related name collision so let's add a related name to 1 of the the namespace collisions so so it really so looking at a model of user has company user user and by the way like my sequel and generated these field and so I'm sort of the terrible have 2 ways the change them but has to foreign keys to user has company and the problem with that that is that the name space for complex so that the accesses to get the results that that will be of will be the same without a generates so battered women into 1 of them

the the announcements the about to again think a database now but so its use and related name told us to do that so was easy enough here's me now to understand me some right so sensor we have that AV-ASR database and now we can run a server

so it work now and it's running so that we have and Florian so I added something

daughter register all the models and so we can look at it right away with look at our company and create a new company so this is weird there's a company ID and I normally don't have enter my on primary keys so let's look at

what went wrong and this is a common thing of the integer field has a primary key but it's not a field and I know that it's not a field because I

created a data model and the company has autoincrement but I can

somehow edit the order implemented value which doesn't really make sense so to fix this

week kitten and our field the

back right so that disappears now the auto increment in the background and security company my schedule not just because that annoys me

I'm sorry for my use of members the now it's user the the

also going on so usernames likely Chris

and my password to the passwords talker the OK so those with the 2 easy tables to map they had 1 primary key there are incremented inmates messed up on identifying it's an auto increment field but that was easy enough to fix and even if we didn't we could assign

a value there of who the next

thing is gonna come up is these composite primary keys so we don't have a primary key listed here so there's multiple ways to approach this

of so related name we talked about

that only have to add my primary keys no also generates from primary key a

composite primary keys of so this always affects this you can drop the primary keys and keep the foreign key constraints and you can add a new field called ID and Janos so like model assumes that there is an ID field anyways and you can make that NOT NULL primary KEY in autoincrement so you're adding this extra fields to each table that has a composite key you just reducing the composite key to this extra field so I

did not memorize how to do that so from the

TV show the so i for the 2 tables that composite keys imatinib call my ID into primary key auto increment and before I did that I dropped the primary keys there existed which were those composite primary keys and then turned to quickly and for the other table for this user's company which the same thing and there some my sequel of book that I have to get around proper foreign keys so I can mean enablers afterwards but for the purpose stomach it doesn't need to be OK so I have a company I have a user and I want put that user in that company without many-to-many relationships so that that come back and cruel role and I know I want to role to the a worker bee the the now inside of my profile from unaccredited profile and given 90 and the work p Prentice misspelled B so we want this role of to be consistent with the role stable even others not foreign key relationship and it's not allow us to do that right now because there's no validation on the role field so that's not going to write a check and see what we can do about that

but

so in our profiles on added validation to ensure that happens and I'm a borrow most of this from here to that of its

foundations are the the import rules so

when import validation Aaron importer roles and if the text for the profile role does not match something that we have in our value list for the names and then we will raise validation romantic so we have 1 profile at that state and then another 1 that's supposedly in a good state or B so worker bee at 1st I misspelling so worker bee is now being validated and it says roles not found so we can but if we change it back to work at B which is in our world so the basically do validation

if something's messed up you can still you know in Python run some check a OK

and answer crater a management command that

kind of demonstrate what you could do but you know this metric man in specific to running on a website so it is wanted to find all the places where you're interact with the database right a code on in Python in in final places where that role relationship is messed up you can do that without text

do not purchase then this whole process the so adamant opposing has a handle it tries to find that apples with the notes doing it look similar profiles and for every profile make sure the role is in the the name

the so if on 1 bad apple in the primary key was 1 the OK so these are the 2 pass I talked

about us so the 1 that talked about you can remove the primary key add a new field named ID and make that the primary key and pre auto-increment field of and the other part

that I didn't specify what was that you can she created you and and change the table name to look that you an increase primary key in that you put it then it messes up your updates inserts also a platform for they took and as other was used to there's a general at that does composite keys I haven't tried out of from that look at that moment the so the runs that he

began what's gonna happen but it's going to generate models for all the stuff that we've already generated source override all the code with it so there's only meant to be a 1 time thing

so if you inspected the epicenter the Inspectorate model for built genomes

right so now the question and this is a trick

question so if anyone wants to break that by liminal will happen if give this code to another developer there were no don't think enough of a so so we have managed at the false

right now the so the angle will create those tables because is unmatched so if we gave this us advances it depends if they have a database that doesn't have those tables are then it's not that create them for them and they're gonna kind of think like what's going on here of and if they have a database that has those tables but then things will work more or less of so that's for that's equal custom command you can if you need to do this database to somebody else you want to create those of files or he gives if you wanna take out the manage forestry the manageable falls apart I

wish I had before it keep talking about the future of but this meant option manageable false so if you wanna take out that manage false than the default values manage equals so if you gave this to somebody else and you took all those out and you menacing TB for that next person would create these tables of and I want to do 1 last being at the head tap actually

and just for fun because this table I created and as we know every table gets mapped model and every column gets mapped to field in Python but what if the fields of would conflict or what they would create things the nonsensical in Django of for example the field name pass which of our car passes a keyword in Python so what's going to happen underscore underscore is going to conflict with the following our foreign key relationships so let's just take a look at what it does just

for fun justifying the field get added with these underscore feel gets added and then since both of these yet it's worth get removed from the next field even though it has 2 underscores and then the 1 underscore field is removed and then they just keep adding these numbers to it so that none of the namespaces Complex I then dear the number 12 is my sequel call name is also invalid in a affair in Python because it's just a number so number underscore 12 it at

but I think I'm at time you sure so

don't write code unless you have too many MIS Chris another Tecumseh High Country friendly guy and you do something which I know of that hoard probably doing it wrong so any academic and my back