Solving Problems with Django Forms
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 19 | |
| Number of Parts | 52 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/32716 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
DjangoCon US 201619 / 52
6
10
13
14
17
20
23
31
36
44
46
47
48
49
00:00
Term (mathematics)Symbolic dynamicsSoftware engineeringValidity (statistics)TrailOperator (mathematics)P (complexity)Computer animation
01:10
Field (agriculture)Validity (statistics)Variable (mathematics)Data structureCore dumpSocial classMaxima and minimaSymbolic dynamicsParallel portElement (mathematics)Electric fieldVolumenvisualisierungContext awarenessAbstractionRule of inferenceSet (mathematics)MereologyMultiplicationCartesian coordinate systemDynamical systemPerformance appraisalArmForceObject (grammar)Fitness functionIntegerComputer animation
03:06
Electric fieldLogicPresentation of a groupElectronic mailing listField (agriculture)Condition numberData structureModule (mathematics)String (computer science)Multiplication signMereologyRight angleVariable (mathematics)Object (grammar)InformationSystem callFilter <Stochastik>Network topologyPerturbation theoryContext awarenessGenderRule of inferenceIntegerSocial classSet (mathematics)Data dictionaryLatent heatSymbolic dynamicsSubsetScaling (geometry)QuicksortVolumenvisualisierungWater vaporOrder (biology)Slide ruleAttribute grammarRun time (program lifecycle phase)QuarkComputer animation
06:50
Element (mathematics)Field (agriculture)Web pageVolumenvisualisierungSymbolic dynamicsMultiplicationAbstractionElectric fieldData structureGenderRight angleTemplate (C++)Bit rate
07:57
Validity (statistics)Symbolic dynamicsElectric fieldAddress spaceRight angleService (economics)ArmFormal languageHypermediaError message3 (number)AreaCartesian coordinate systemOvalCASE <Informatik>
09:00
Error messageGenderElectric fieldWeb applicationCASE <Informatik>Field (agriculture)System callRight angleAreaElectronic visual displayGoodness of fit
09:58
Field (agriculture)TrailElectric fieldFunctional (mathematics)Vector potentialCausalityMereologyoutputData structureComputer configurationSymbolic dynamicsElectronic mailing listEuler anglesAdditionBitComputer animation
11:32
MereologyField (agriculture)Electric fieldDrop (liquid)ProgrammschleifeRight angleState of matterProcess (computing)Validity (statistics)Data dictionaryElectronic mailing listWeb pageImplementationMathematical analysisMathematicsContent (media)Point (geometry)Data structureBitTemplate (C++)Context awareness
13:06
Field (agriculture)Data storage deviceData structureState of matterElectric fieldTemplate (C++)Software frameworkDecision theoryCartesian coordinate systemConnectivity (graph theory)VolumenvisualisierungBitWeb pageRegular graphQuicksortForcing (mathematics)Computer architectureSingle-precision floating-point formatSymbolic dynamicsGenderDebuggerFront and back endsComputer animation
14:47
Multiplication signMereologyState of matterValidity (statistics)Electric fieldType theoryHuman migrationQuicksortWeb pageFront and back endsField (agriculture)TrailoutputService (economics)Library (computing)AdditionDebuggerData structurePlastikkarteSystem callError messageCuboidMessage passingInheritance (object-oriented programming)Open sourceSet (mathematics)Server (computing)NumberLevel (video gaming)Scripting languageString (computer science)CASE <Informatik>VolumenvisualisierungProjective planeTraffic reportingComputer virusOpen setComplex (psychology)Connectivity (graph theory)Term (mathematics)Point (geometry)INTEGRALCross-site scriptingInformation securityContent (media)MathematicsCanadian Mathematical SocietyWordMatching (graph theory)Instance (computer science)Motion captureOrder (biology)Constructor (object-oriented programming)Right angleElement (mathematics)CausalityMathematical analysisLatent heatCellular automatonResultantAreaPerformance appraisalSubsetCovering spacePressureComputer animation
23:10
Computer animationJSONXML
Transcript: English(auto-generated)
00:00
Come on! So before I get started, just to raise hands, how many people have heard of or worked with crispyforms?
00:22
OK, so there's a lot of people. I want to see how many people are going to be upset at me if I say things about crispyforms before I get started. So quick, just to introduce myself, I'm Kurt Gittens. Like you said, I'm a software engineer at DealerTrack. And at DealerTrack, we work a lot with JingleForms.
00:42
We have a lot of situations where we need really complicated data entry, a lot of dynamic validations and dynamic forms. I'll kind of get into what I mean by dynamic forms. But this talk comes from a lot of what we've been building at DealerTrack and what we've learned in terms
01:01
of how to kind of use JingleForms to extend its capabilities to be dynamic. But before I go into the problems that we actually solved, I want to kind of back up and talk about what JingleForms actually is. And this is kind of parallel to what's in the Django documentation. But basically, JingleForms offers you abstractions
01:21
over three kind of core things. So you get an abstraction over the structuring of a form, which tells you what fields are going to be on the form when you write your form class. You define these class-level variables that are objects, that are your fields. So the structure of your form is basically that, like what fields are going to be on the form.
01:41
Then there's the rendering of a form, which is like Django takes care of actually creating the HTML elements that your users need to actually put data into. And so Django takes care of that for you. There's a template tag. All you need to do is use the form tag, and it renders the form. So JingleForms gives you functionality for that. And the last part is validating and processing
02:02
the data. And that is where you define a set of rules for when the data that the user enters is going to be accepted. So you have your, the basic Django validations that you get from like validators, that these are things that Django adds. Like when you create like an integer,
02:22
or fields that capture like numeric data, and you set a max and min value, you have those kind of validations, and then you have more complicated validations that you can write in your form clean, or your individual field clean. So these are kind of the three main things that Django gives you. And so a lot of the situations that we're having at DealerTrack, we needed to create dynamic.
02:41
We need to basically make all three of these things dynamic. And so starting with structuring, what I mean by creating a dynamic form structure is that we needed, we had a situation where we needed a form where the fields on it might change depending on certain pieces of user context. And this is like probably a problem that multiple,
03:02
it's not just unique to us, right? A lot of people might have to deal with something like this. So like the reason why you might need to do something like this is like, say you have a really basic form that captures some data, but then you need to introduce a piece of context to change the form, right? So you can do that like this, right? You can add an if condition that adds a field to the form,
03:22
right? Self.fields is a dictionary. You can put a field into it dynamically when you instantiate the form, and everything works. The problem with this is that, oh, hold on. Yeah, that's the right slide. So the problem with this is it doesn't scale when you have a lot of conditions. So when you start getting really complicated layouts
03:41
that have a ton of conditions and a ton of fields that need to change, this solution doesn't work too well because you'll have a really messy form in it that like has a ton of if conditions and a bunch of custom business logic. And what you want to do is separate the two of those things, right? You want to be able to separate your logic that determines what you see as part of the form structure
04:01
from those rules. And so the way that we chose to solve this problem is by treating our fields as data. And what that means is like the way that you define fields with Django right now is you use, like, it's an object, right? And you set some quarks on an object when you instantiate it.
04:20
And that determines what Django renders and how Django validates it. But all that information you could basically represent as a dictionary, right, instead of an object. And working with it as a dictionary makes it a lot easier to kind of move the layouts around. Your data becomes a lot more malleable.
04:40
So we start off with a solution that looks something like this, right? So your fields now become like entries in an ordered dictionary. And your field structure is no longer directly tied to the form. Instead of having a class definition that says, here's the fields. Here's the order. You have this, which basically takes in your field structure object, which is basically the same as what self.fields
05:03
ultimately builds. But what you get now is you have a layout that's separate. So if you want to have a different layout, you define a different variable with a new ordered dictionary. And you can actually take this a step further and move the actual field objects out of it, or rather these field variables with the data,
05:21
and kind of replace them with the way this solution works is so you now, instead of having a ordered dictionary that contains the actual names, you create a list of strings. And you grab those names from some sort of module. In this example, I'm using a class. But you could swap that out for anything, right?
05:42
So you have a list now that defines your field layout. And it's completely separate from your Django form. So what you can do, and what we actually do at DealerTrack, is we have an API call that determines what fields are on the form at runtime. So our API takes care of the context and all of the specific business rules
06:02
that determine what fields need to be on the form. And then it spits out just a list of strings. And this knows what to do after it has that list of strings, right? It does a get attribute on the class that contains the field. So we have this kind of class that will contain all the possible fields
06:21
that could be on the form. Since you need to have a superset, you need those fields definitions somewhere. So we grab an attribute from that class. We basically instantiate the actual Django field object, whether it's a character field or an integer field or something like that. And then we just stick it in the self.fields
06:40
because we can add entries to that really simply. So that's all you need to do to have a dynamic field layout. But the other piece of it is actually rendering this. So we need to get HTML from this dynamic field layout. So we know about the Django form tag, right? You just place it into your template, and it renders all the fields on your form for you.
07:02
One of the problems with this, though, is if you need to have any custom HTML or CSS instead of what Django generates for you, it becomes difficult because you're not exactly sure what elements are going to be on the page. So for a while, we had a solution that worked with Django Crispy Forms.
07:20
But I think for stuff like this, I would advise staying away from Django Crispy Forms because it forces you to tie your particular form to a particular layout. So Crispy Forms gives you abstractions over the actual HTML elements that you'll see on the page. But if you have a dynamic layout, you
07:41
don't really want to tie it up directly to your form definition, right? You could use that layout in multiple forms. And you don't want to be tied to one particular rendering of that. So that's kind of how the dynamic field structure problem works. So one of the other problems that we had to solve was dynamic form validation.
08:02
Kind of an example of why you would want to do this, kind of a really trivial one, is if you have an address field in your application, you might want to call out to a third party service to validate that your address is actually correct, right? But the problem is that third party service has errors. Your Django form has errors. And you want the two of those to really act as one.
08:23
Because it's just interesting to talk about, our actual use case at DealerTrack was that we have third party users who write validations in a language that we've created that influence our Django form. And then those validations that they write are evaluated by a microservice that we have.
08:42
And that aspect of it is actually pretty complicated. But Django allows us to kind of not worry about dealing with those errors once we actually get them back from the API. So all you need to really do to integrate external errors that you get from another service with the internal errors of your form is just call add error.
09:01
So I wouldn't actually call clean, or call an API directly in clean like this. But basically, the approach is kind of the same, right? You make a call out to an API. You map up the errors that you get back from the API with the fields that you have in your form.
09:20
And then you just call add error. And Django takes care of the rest. So what's good about this is if you have an existing infrastructure for displaying errors to the user like I'm sure a lot of web applications do, you need your user to see in a lot of cases what's wrong on the form. This allows you to integrate those external errors as if they came from Django. And you don't have to worry about doing
09:41
any of the extra work. So you just add error takes a field and an error message and it displays it on that particular field. Or it adds it to the field in the Django form and then however your display works is gonna continue to work the same way. So one last problem that we looked at at DealerTrack was having user-driven fields.
10:02
And basically, what I mean by this is you give the user ability to add a field to your Django form. So kind of to look at an example of this, right? So this is in the middle of the animation. But basically, this demonstrates like say you have a list of potential optional inputs
10:23
that you want the user to be able to add, the user can select one and add it to your Django form. So it seems like at first there might be a lot of problems because we think of Django forms as like you have a static definition of the form. But you can actually handle this completely
10:40
with our previous solution. Obviously, you need a little bit of JavaScript to kind of make sure that this works the way that it is supposed to work. But there's basically like three really simple pieces to the solution. So on the UI aspect, actually rendering the fields is taken care of by JavaScript. So which might seem a little strange because you'll have initially a mismatch between what
11:03
the user actually is seeing and data entering and what is actually on your Django form when you render it. But you allow JavaScript to basically implement the functionality to allow the user to add the fields to the form. And then when you save, your dynamic field structure part
11:20
that we talked about before comes into play because now you can change, you can add those additional fields that the user added to the form. So the JavaScript part is not that complicated. It looks kind of like, this is like a really trivial implementation of it. And basically, the key part is that you need to pass data from your template context
11:41
in when you're rendering the Django template to JavaScript. So like here, I have the drop down contents, which is like a list of all the potential fields I want to allow the user to add. And then save drop down fields is a dictionary of the field, if the user has saved data for it already, and the value
12:01
that the user has saved for it. The reason why I need both of those things is because I don't want the user to lose any data. So if they save something, when they reload the form, JavaScript needs to create the fields that they've already saved. So all this does is it loops through the drop down contents, and if the user saved data for it,
12:21
it creates like a static field that looks like everything else. If the user hasn't, it adds it to the drop down. And then what actually drives the drop down is just like a really simple JavaScript that checks for a change and adds a field in the same way that this might. The Django form side of it is even simpler because we have our base fields, which
12:41
are what you see when you render the form. And then you have the drop down fields. And now when you're saving, you can just add the drop down fields into your field structure because you can change it at any point. You can load with a different form than you save with. And now your validations work even though those fields weren't in your form when the user loaded the page.
13:03
So one last thing I want to talk about a little bit is a lot of these problems ended up leading us to the decision to move away from the architecture that we currently had, which was Django forms and just regular HTML Django templates.
13:20
And we moved to React.js to Redux. But the actual back end is pretty much the exact same in React.js and Redux. We still use our dynamic field structure. And that actually helps us more when we move to React.js because what happens is we implement React components that
13:42
mirror the Django fields. And then what happens is you send your whole field structure, instead of letting Django render it, you can dump it as a JSON and treat it as data once again and send it to the front end and let React render it. And then React kind of treats it. You can treat your forms then as pages in your single page
14:03
application, or yeah, single page application framework with React. And then what we use Redux for is to kind of imitate this sort of state storage and save that you would get with doing regular post requests. So Redux is just like a plug-in that you can use that allows you to store state.
14:22
React just handles the actual rendering. So Django form spits out a field structure, the same thing that we had before. React renders it because you have the same React components that mirror the Django fields. And then Redux just takes care of making sure that the data that the user enters is saved. And then when the user wants to post back to the form,
14:43
they can do that via Ajax. So that's really it. Any questions? I think I have some time for questions and answer. Sorry about the last part about using React and Redux. How do you handle the server side validation?
15:04
You usually get messages from the back end. How do you present those into the React UI? So the way that this actually works is we're kind of, and this solution is not in the best stages that it currently could be. But our back end basically instantiates a Django form
15:25
because we have that field structure which represents a Django form. So the back end instantiates a Django form with the data that it gets via Ajax from React. We get form errors back. And then we just send those form errors back as JSON to React JS.
15:40
And then React renders it the same way that we would before. Does that answer your question? Thanks. Do you want to go? All right. Thanks for the talk. That was really interesting. In fact, I work with a Django-based CMS called Wagtail that's very similar in the way
16:01
that it's handling its own forms and its admin. Do you have any use cases where you have saved the state of the form at the time that it rendered? Because if the form is just data, then you could save that JSON string of that dictionary and then access that form in the exact state as it was during that time.
16:21
And then you kind of have this sort of Django, like the migration state type thing with your forms. So with Redux, that actually kind of is what you get. So Redux tracks the state of the form. So as soon as you render the page,
16:40
getInitial populates the field values. And then that also populates Redux's state. So at that point, you have a state that basically represents all of the data that you have on the page. And that state stays updated. So if the user changes something on the UI element, it updates in the Redux state. And then that's kind of how we don't do anything else
17:03
to capture the post data. Redux just stays updated with the state. And then when you save, or when you post back to the server via Ajax, it's the same state. And whatever the user updated gets updated when you send it back. So I really love the idea of sending the form structure
17:24
to React as JSON and having React components that mirror the Django form structure. So my question is just, are your React components for forms open source? Or do you know of any available projects that have fields that mirror the Django form structure?
17:42
That is a good question, because we really should. And I think that is an ultimate goal of ours, is to open source the solution once it becomes detangled from all the weird special case stuff that we have written in it. So that's probably something that's going to happen in the future. Because I can see that being useful to a lot of people who
18:01
want to integrate Django and React is, if you want to go that route, we already have a solution built. And I want to be able to say that it's going to be open source at some point in the near future. I just can't tell you exactly when that would be. I don't know if there are other projects that have that sort of thing. We can't be the only people integrating Django and React.
18:22
And I think there's another talk even happening about that integration. So I'd be curious to see what other libraries are out there. I don't really know. Great, thanks so much. Hey, you mentioned you have sometimes third party services where you do validation on certain user inputs.
18:40
And you don't want to do the actual appending of the errors in the forms clean method. So where do you actually do that? Or what's your approach there? So I meant doing the service call in the actual form clean. What I would just do is move that out into a method so you don't have this huge, because you're
19:00
going to have a whole bunch of stuff in your clean if you have any sort of complex requirements as far as the data that can be entered. So yeah, in terms of where you would add the third party errors, I would just do it in another method on your form that gets called in clean. But make sure that your clean doesn't become some unmanageable thing is kind of what I was getting at with that.
19:24
Hello. Thank you for your report. I have a question. So with your approach, front end controls the structure of the form, right? So are there some security issues like cross-site scripting? Like from JavaScript, I can post some unwanted fields
19:43
that shouldn't be there? So I guess the content does not control the structure of the form, right? The data, I guess you're talking about more the user-driven fields thing. And even that, there always has
20:00
to be a super set of what is allowable, because you can't let the user define custom fields. Or if you do, you kind of have to hack around it and do something where the custom field that the user is defining is actually some, it's a concrete field on the back end, you can never let the user modify your actual form structure. So what we have is we have a super set of all
20:21
the possible things the user could do. We don't let them add their own dynamic things that aren't part of our super set, because you're right, that would have security issues. I mean, even so, there is, for example, a field like credit card number. And you expect it only in some circumstances, like user should fill in his first and last name
20:42
or some check box. But I can construct JSON that contains credit card without those fields. And I can send it. So do you have some additional server side validation for such cases? Yeah, so the field structure, the form that renders,
21:02
it has to match, or rather, the form that the user enters data into has to match the back end. They're not two separate structures. So if I send you, if I render a field structure, but then the user enters an additional, I guess in that case, that doesn't really hit the server side, if I understand
21:21
what you're suggesting. Is that like somebody scripts an additional field that isn't really on the form? It is on the form, but it appears only in some case. Right, and in that case, your field structure on the back end would know that that's not there. And so that data, you would get some sort of validation
21:40
error on the back end. When you try and post a field that isn't there, that would cause a problem. OK, thanks. Have you tried to allow a user to get halfway through a kind of long form and save their state and then come back to it and pick up where they left off, maybe in the middle of one particular form?
22:02
Yeah, so we've gone back and forth about that, it seems like. But yeah, do you have some other specific questions about doing that? Just yet, have you had challenges, since you have tried it, have you had challenges integrating that, doing that with this workflow? Yeah, the problems with that, we
22:21
have had a lot of challenges with that. The problems with that tend to be validation. We have all these problems about what kind of validation. Some validations are not important all the time. For instance, in that case, if you want the user to be able to save partial data and come back later, you don't want to always run your required validation.
22:41
But your required validation is important, so we've gone back and forth about what you actually do in those circumstances. But I think it's definitely, with this structure, it's not a super difficult problem to solve. I think you can go back and forth pretty easily on what you actually want the end user behavior to be.
23:03
So I think that's it then.