From Legacy to Admin
Formal Metadata
Title |
From Legacy to Admin
|
Title of Series | |
Part Number |
26
|
Number of Parts |
44
|
Author |
|
Contributors |
|
License |
CC Attribution - ShareAlike 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license. |
Identifiers |
|
Publisher |
|
Release Date |
2014
|
Language |
English
|
Content Metadata
Subject Area | |
Abstract |
For those who have not had the pleasure of seeing django's inspectdb command in action, I will create a demonstration of it's power. Django's inspectdb command can reverse engineer a set of models from a postgres or mysql database. I will demonstrate how to take a legacy database and create a quick and dirty admin tool along with a simple rest interface.
|

00:00
Computer animation
Model theory
Reverse engineering
00:40
Information
01:01
Computer animation
Information
Internet forum
Code
Gender
Right angle
Mereology
Writing
01:54
Mobile app
Standard deviation
Process (computing)
Code
Multiplication sign
Model theory
Database
Table (information)
Row (database)
Physical system
Social class
02:35
Programming language
Key (cryptography)
Validity (statistics)
Sequel
Relational database
State of matter
Interface (computing)
Gradient
Model theory
Database
Field (computer science)
Latent heat
Computer animation
Personal digital assistant
Customer relationship management
Resultant
Physical system
Row (database)
03:41
Computer animation
Demo (music)
Code
Source code
Database
Physical system
04:01
Computer animation
Demo (music)
Code
Operator (mathematics)
Interface (computing)
Source code
Right angle
Database
Product (business)
Physical system
04:39
Scripting language
Computer animation
State of matter
Network topology
Model theory
Statement (computer science)
Set (mathematics)
Database
Endliche Modelltheorie
Table (information)
Metadata
05:19
Point (geometry)
Group action
Electric generator
Key (cryptography)
Inheritance (object-oriented programming)
Gender
Multiplication sign
Keyboard shortcut
Model theory
Fitness function
Field (computer science)
Database normalization
Computer animation
Personal digital assistant
Profil (magazine)
Self-reference
Pattern language
Table (information)
Error message
07:36
Functional (mathematics)
Sequel
Code
Source code
Electronic mailing list
Database
Representational state transfer
Function (mathematics)
Mereology
Connected space
Type theory
Subject indexing
Goodness of fit
Latent heat
Loop (music)
Process (computing)
Computer animation
Software testing
Table (information)
Oracle
09:19
Loop (music)
Computer animation
Gender
Order (biology)
Electronic mailing list
Selectivity (electronic)
Table (information)
09:58
Subject indexing
Computer animation
Information
Mapping
Relational database
Endliche Modelltheorie
Table (information)
Field (computer science)
Connected space
10:27
Point (geometry)
Complex (psychology)
Mobile app
Sequel
Computer file
Function (mathematics)
Field (computer science)
Mathematics
Computer configuration
Endliche Modelltheorie
Mapping
Key (cryptography)
Namespace
Relational database
Gender
Model theory
Database
Line (geometry)
Computer animation
Order (biology)
Collision
Quicksort
Table (information)
Reading (process)
Resultant
Directed graph
13:26
Server (computing)
Computer animation
Database
Right angle
14:07
Computer animation
Key (cryptography)
Integer
Field (computer science)
14:31
Data model
Scheduling (computing)
Computer animation
Order (biology)
Information security
Field (computer science)
15:00
Computer animation
15:19
Computer animation
Key (cryptography)
Password
Table (information)
Field (computer science)
15:56
Computer animation
Key (cryptography)
16:15
Key (cryptography)
Relational database
Model theory
Table (information)
Field (computer science)
16:50
Computer animation
Key (cryptography)
Sequel
Validity (statistics)
Profil (magazine)
Right angle
Table (information)
Field (computer science)
18:30
Computer animation
Validity (statistics)
Profil (magazine)
Rule of inference
19:01
Goodness of fit
Computer animation
Validity (statistics)
Profil (magazine)
State of matter
Electronic mailing list
20:00
Existential quantification
Computer animation
Code
Customer relationship management
Interactive television
Website
Database
20:33
Message passing
Process (computing)
Computer animation
Key (cryptography)
Profil (magazine)
21:00
Computer animation
Key (cryptography)
Moment (mathematics)
Insertion loss
Mereology
Table (information)
Field (computer science)
Computing platform
21:32
Computer animation
Code
Multiplication sign
Source code
Model theory
Function (mathematics)
Endliche Modelltheorie
21:52
Computer animation
Computer file
Angle
Code
Software developer
Equaliser (mathematics)
Database
Table (information)
22:47
Default (computer science)
Message passing
Computer animation
Model theory
Disk read-and-write head
Table (information)
Field (computer science)
23:45
Computer animation
Sequel
Namespace
Multiplication sign
System call
Field (computer science)
Number
24:24
Computer animation
Lecture/Conference
Code
00:00
in a in a year and a half and I move around a lot is lazy and that you guys are coming up of know you add many other options so that the end of this talk and I hope to be informative and about inspect each of which is a general comment that helps you of reverse engineer model from an existing database on
00:41
within Inspector Chris for all but I go by monodomain Chris of when I was practicing this I ran along some Fig kind of fast to trying as much information as they can on this is me there's a
00:55
professional Christian and the regular chris so that you can talk to me afterward so you know your
01:01
talk to graduate not professional cross but right now
01:05
it's a professional cross on so something to the talk that I wanted to give our don't right so don't write code unless you have 2 and my name is chris not data come up and if you do something to do something hard which I know that you probably doing it wrong and this and is something I realize I started using Django initially on when I say so when I created a custom command to populate initial data and I realize that there were fixtures of so this happens a lot when you like when you start to work on something you don't read all the documentation or you see some cool part of it you wanna do not realize that there's other things of so my advice is you know to spend as you come the general con and go of Gaia Iousy of gold forums and there's a lot of rich information there's Fletcher such gender and that's and there's a lot of like custom
01:56
apps uh templates TL have like their suggested ways of doing things so inspected B is 1 way to save a lot of time and if you work
02:05
with legacy systems so instead the reads tables and columns not rows in a database and produces a model stop I've standard out of have to manually write this code and you could you could well you know create a class that inherits from all the model and give an explicit TV DVD it like mean and and you could go through every table in your database but this just jump start the process for you it's not perfect so we'll go over why what what we got do afterwards but
02:35
it works pretty well and when you need this I and 2 scenarios have come up new this idea that I have legacy legacy system listed in this Annex 1 admin interface to it of so foreign keys are grade of the primary key constraints a great result mean always and strengthen my sequel comes with but in reality you have like an additional of validation layer to let the system that you want keep it clean state you can you reverse-engineer model at a couple validations and you'll have this slicker admin interface of the general comes with for free on the other use case that I've come up
03:11
with is a pretty management commands us so when you create management commands for going through much of rows of a database and making sure that 1 field is less than another field or you won't find all of rows that have you know a certain value the or em comes in handy for 2 for doing these things and writing custom sequel is sometimes hard of particularly if like you know it's not not designed to be a programming language designed to specific things in sequel that's another reason why this is come handy in the past so
03:43
credit created a fictitious scenario of y of why I'm doing this demo and I have a legacy system with a legacy databases and I can't touch the code for whatever reason I and I need to be able to manipulate data but still keeping a clean state and so I can have the source and there's a
04:03
lot of reasons why you India was at the source you have you have a source of to third-party product it's written rails or of you know the source principle it's impossible understand also at the end of this demo I
04:19
want I want to be able to have an admin interface that's there was little work as possible and you will have a basic right operations and 1 be a little leave the existing database intact more or less and we'll talk about how we break that when we have to break that so many ways to ways to get around it but a more or less what that leads the system it's interacting with this code to to be the same after
04:39
this also the tree from the docks
04:44
insect avait and database and whatever's pointed so the 1st thing you'll notice here is that of whatever your settings of name is pointed to death with can introspect of and it's also in this state the statement doesn't seem like it says a lot but it actually says a lot the script once metadata asynchrony model for each table on so something that you don't think about is that you often have more tables than models because of many to many relationships and inspected the inspection don't don't understand the concept of and also the most important part of this is this
05:22
feature is meant to be a shortcut not is different among generations so I am I created a
05:28
fictitious model yeah so it's pretty
05:33
basic there's a user table I made a lot of like egregious errors don't blame me I designed this to be terrible on purpose so that we could understand what would happen if you designed the is not yet a work with is the terrible manure expecting them they're not to be perfect so every user table it has 1 primary key it's our incremented it's not null and has username and then of course it has a point X that's work of we have a company table has 1 of the primary key is Otto incremented if that in a minute self-referential so a company can have a parent company and so it's interesting corner case and then this goes on before a user can belong to many companies and many come and a company can have many users and their related by this user has company of many many table so this relationship in in uterine Django you have a user model that has a many-to-many relationship or a company or vise versa but there's really no way of of work of gender inspection to know this so we're missing at what this action produces a 2nd and another thing that's terrible but this referring to the Django at least that is that has 2 primary key so has composite primary keys which is gonna be a problem for us some of the y the of Agenda basically since the you have every single model this in the of 1 ID field that has a primary key and this is doesn't fit that pattern and then profile budgetary thought community worse has a triple a composite key and there's a role here so role is meant to be a foreign key to this table role but this is of our car and the name years so as to match the role in here there's like completely worst interests and you can imagine there can be so many problems with this obser anomalies deletion anomalies like you name it so that's but smashing model and then just for fun if we have time at the end and go over this table because I created like the worst of field names possible just to see what they would produce and so the source of inspect the
07:40
be all and this is the source of new it's good size I guess so that there's 2 primary lives the 1st loop other users connection that infection table names also connection introspection is a of connection specific introspection function to get the table names so depending on what database you're using it'll get it'll user differ command to get back at list of table names but then once it has that was the table names of its agnostic this code of the rest of goes agnostic to the type of database you have so I looked at the source code for other the different introspection definitions so my sequel oracle and posters because all of them and you look at the documentation it says my superimpose grows over pretty much was before spices and and oracle has the same function introspection functions methods defined to introspect database so I don't get a chance to test oracle but I don't see the reason why when work and I was looking online it seems like a lot people driving to work so maybe I can look up that later in the talk about that but so once you get a table name and at that 1st primary loop the secondary part of the loop of the inside that loop I guess relationships gets indexes and then it has a secondary loop to loop through all the other columns and at the end of this and ardently do the job of the screenshot but the end of this it generates a collocate yields a the output of to the use an output of the column name to standard out so that already used so this was a little quiz I made
09:20
a these are the intersection Pacific commands that my seat so that gender uses that to get table names so my he goes to show tables are select tables for Oracle and something knows about these these each of these commands is not guaranteed to come in any particular order social tables just gives you a list of the tables of it doesn't give you the table sorted by you know when they were created and use the table sorted by of the name and all these are the same so we go through that loop it's basically going get up and there's just no way to guarantee what order it but so this is
09:58
what we talked about before the we use the connection introspection about to get the table names relation of new indexes and the columns but then we have agnostic at the next layer to generate the models once we have all information from our a connection and this is the many-to-many fell of
10:16
like I said this should map with a many to many on the the use of the company field depending but this maps 3 individual models so let's take a look at
10:28
what would happen if we do this so Python managed to apply it to the show so I created this in my sequel so we'll take a look at us so these are the 6 tables we started with and we haven't think the database so we don't have any of the gender-specific of tables so the question is do we wanna singer database or do database introspection 1st the database introspection looks to the database and create a model from the DB creates ah since a database of to the correct point of that were out right now or whatever we have the and and uh right now so if we run our sink database 1st order create tables that of gender needs to be run at to log in and you can't older users but then is we run uninspected days after that those tables will also be in our database and they will be introspected as well and that's not really what we want because those are those models already defined somewhere else around them inspect B 1st the so like I said goes directly standard out of which is not very useful but you can take a look at it of things so tells you to do things which is nice so I had to be be giving you talk industry this you rearrange the models you make sure that and everyone has a primary key and if you want you can remove managed falls on the doesn't really say with that's gonna do besides its can allow that gender to have a lifecycle and allowed to rename the models but you're not allowed to read DB tables which is like the Met option to explicitly state the table name is to override the convention of and then there's this line which you know I'm sure won't be important all you have to insert the output of gender and sequel custom app name your whatever so let's go ahead and put this to so I created map for this you have and let's assume we
12:36
produce this is same thing but worse in a file now of and now we can run our safety the the so the 1st from the run into is of related name collision so let's add a related name to 1 of the the namespace collisions so so it really so looking at a model of user has company user user and by the way like my sequel and generated these field and so I'm sort of the terrible have 2 ways the change them but has to foreign keys to user has company and the problem with that that is that the name space for complex so that the accesses to get the results that that will be of will be the same without a generates so battered women into 1 of them
13:27
the the announcements the about to again think a database now but so its use and related name told us to do that so was easy enough here's me now to understand me some right so sensor we have that AV-ASR database and now we can run a server
14:02
so it work now and it's running so that we have and Florian so I added something
14:09
daughter register all the models and so we can look at it right away with look at our company and create a new company so this is weird there's a company ID and I normally don't have enter my on primary keys so let's look at
14:23
what went wrong and this is a common thing of the integer field has a primary key but it's not a field and I know that it's not a field because I
14:32
created a data model and the company has autoincrement but I can
14:38
somehow edit the order implemented value which doesn't really make sense so to fix this
14:43
week kitten and our field the
14:51
back right so that disappears now the auto increment in the background and security company my schedule not just because that annoys me
15:03
I'm sorry for my use of members the now it's user the the
15:19
also going on so usernames likely Chris
15:38
and my password to the passwords talker the OK so those with the 2 easy tables to map they had 1 primary key there are incremented inmates messed up on identifying it's an auto increment field but that was easy enough to fix and even if we didn't we could assign
15:57
a value there of who the next
16:00
thing is gonna come up is these composite primary keys so we don't have a primary key listed here so there's multiple ways to approach this
16:08
of so related name we talked about
16:12
that only have to add my primary keys no also generates from primary key a
16:17
composite primary keys of so this always affects this you can drop the primary keys and keep the foreign key constraints and you can add a new field called ID and Janos so like model assumes that there is an ID field anyways and you can make that NOT NULL primary KEY in autoincrement so you're adding this extra fields to each table that has a composite key you just reducing the composite key to this extra field so I
16:49
did not memorize how to do that so from the
16:54
TV show the so i for the 2 tables that composite keys imatinib call my ID into primary key auto increment and before I did that I dropped the primary keys there existed which were those composite primary keys and then turned to quickly and for the other table for this user's company which the same thing and there some my sequel of book that I have to get around proper foreign keys so I can mean enablers afterwards but for the purpose stomach it doesn't need to be OK so I have a company I have a user and I want put that user in that company without many-to-many relationships so that that come back and cruel role and I know I want to role to the a worker bee the the now inside of my profile from unaccredited profile and given 90 and the work p Prentice misspelled B so we want this role of to be consistent with the role stable even others not foreign key relationship and it's not allow us to do that right now because there's no validation on the role field so that's not going to write a check and see what we can do about that
18:30
but
18:33
so in our profiles on added validation to ensure that happens and I'm a borrow most of this from here to that of its
18:44
foundations are the the import rules so
19:11
when import validation Aaron importer roles and if the text for the profile role does not match something that we have in our value list for the names and then we will raise validation romantic so we have 1 profile at that state and then another 1 that's supposedly in a good state or B so worker bee at 1st I misspelling so worker bee is now being validated and it says roles not found so we can but if we change it back to work at B which is in our world so the basically do validation
20:02
if something's messed up you can still you know in Python run some check a OK
20:11
and answer crater a management command that
20:13
kind of demonstrate what you could do but you know this metric man in specific to running on a website so it is wanted to find all the places where you're interact with the database right a code on in Python in in final places where that role relationship is messed up you can do that without text
20:34
do not purchase then this whole process the so adamant opposing has a handle it tries to find that apples with the notes doing it look similar profiles and for every profile make sure the role is in the the name
20:54
the so if on 1 bad apple in the primary key was 1 the OK so these are the 2 pass I talked
21:03
about us so the 1 that talked about you can remove the primary key add a new field named ID and make that the primary key and pre auto-increment field of and the other part
21:12
that I didn't specify what was that you can she created you and and change the table name to look that you an increase primary key in that you put it then it messes up your updates inserts also a platform for they took and as other was used to there's a general at that does composite keys I haven't tried out of from that look at that moment the so the runs that he
21:35
began what's gonna happen but it's going to generate models for all the stuff that we've already generated source override all the code with it so there's only meant to be a 1 time thing
21:47
so if you inspected the epicenter the Inspectorate model for built genomes
21:53
right so now the question and this is a trick
21:56
question so if anyone wants to break that by liminal will happen if give this code to another developer there were no don't think enough of a so so we have managed at the false
22:10
right now the so the angle will create those tables because is unmatched so if we gave this us advances it depends if they have a database that doesn't have those tables are then it's not that create them for them and they're gonna kind of think like what's going on here of and if they have a database that has those tables but then things will work more or less of so that's for that's equal custom command you can if you need to do this database to somebody else you want to create those of files or he gives if you wanna take out the manage forestry the manageable falls apart I
22:47
wish I had before it keep talking about the future of but this meant option manageable false so if you wanna take out that manage false than the default values manage equals so if you gave this to somebody else and you took all those out and you menacing TB for that next person would create these tables of and I want to do 1 last being at the head tap actually
23:14
and just for fun because this table I created and as we know every table gets mapped model and every column gets mapped to field in Python but what if the fields of would conflict or what they would create things the nonsensical in Django of for example the field name pass which of our car passes a keyword in Python so what's going to happen underscore underscore is going to conflict with the following our foreign key relationships so let's just take a look at what it does just
23:47
for fun justifying the field get added with these underscore feel gets added and then since both of these yet it's worth get removed from the next field even though it has 2 underscores and then the 1 underscore field is removed and then they just keep adding these numbers to it so that none of the namespaces Complex I then dear the number 12 is my sequel call name is also invalid in a affair in Python because it's just a number so number underscore 12 it at
24:20
but I think I'm at time you sure so
24:28
don't write code unless you have too many MIS Chris another Tecumseh High Country friendly guy and you do something which I know of that hoard probably doing it wrong so any academic and my back
