A Multi-submission importer for EasyForm

Video in TIB AV-Portal: A Multi-submission importer for EasyForm

Formal Metadata

A Multi-submission importer for EasyForm
Title of Series
N. N. (Moderation)
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Release Date
Production Year

Content Metadata

Subject Area
I needed a way to mass import form submissions into Plone’s EasyForm, so after a bit of exploration and some creativity I built a custom CSV import tool. Through an Action on the tool-bar, a user can export a CSV with the appropriate form fields, update it, and import back to that form. Each row represents an EasyForm submission and mimics the submission process for that form. Even the form actions get processed. In this talk, I will walk through the my process examining how EasyForm handles submitted information, following the trail deep into CRUD forms, and piecing that newfound knowledge together to create the tool.
Plane (geometry) Slide rule Zoom lens Multiplication
Multiplication Software developer Multiplication sign Software developer View (database) System administrator Mass Image registration Client (computing) Mass Instance (computer science) Field (computer science) Revision control Event horizon Different (Kate Ryan album) Website Website
Group action Demo (music) System administrator Multiplication sign System administrator Mass Client (computing) Image registration Group action Template (C++) Template (C++) Website Website
Web page Wechselseitige Information Group action Email Electronic data interchange Electronic data processing Multiplication sign Image registration Image registration Template (C++) Field (computer science) Template (C++) Annulus (mathematics) Intrusion detection system Normed vector space Right angle Row (database)
Zoom lens Group action Email Process (computing) Computer file Personal digital assistant View (database) Right angle Hidden Markov model Group action
Graphical user interface Process (computing) Disintegration Cloning Group action Library (computing)
Web page Point (geometry) Axiom of choice View (database) Code Computer file Image registration Client (computing) Template (C++) Web browser Field (computer science) Order (biology) Interface (computing) Information Gamma function output
Module (mathematics) Context awareness Trail Email Computer file Dependent and independent variables Multiplication sign File format Field (computer science) Insertion loss System call Timestamp Field (computer science) Mathematics Personal digital assistant Order (biology)
Group action Graphical user interface Process (computing) Sample (statistics) Computer file File format Function (mathematics) System administrator Website
Web page Computer file View (database) Multiplication sign View (database) Computer file Interface (computing) Social class Information Image registration Template (C++) Web browser
Point (geometry) Inheritance (object-oriented programming) Inheritance (object-oriented programming) Code Factory (trading post) Code Damping Endliche Modelltheorie Template (C++)
Context awareness Information Inheritance (object-oriented programming) View (database) Plastikkarte Group action Mereology Web browser Product (business) Template (C++) Computational complexity theory Personal digital assistant Motion blur Website Information Data structure Row (database) Social class Address space
Context awareness Slide rule Information File format View (database) Uniqueness quantification Code Field (computer science) Right angle Web browser Descriptive statistics
Boolean algebra Tuple Zoom lens Functional (mathematics) Slide rule Computer file File format Code Multiplication sign Set (mathematics) Field (computer science) Bit Electronic mailing list Drop (liquid) Field (computer science) String (computer science) Source code output Cuboid Control theory Selectivity (electronic) Figurate number Social class
Web page Context awareness Mathematics Computer file Dependent and independent variables Personal digital assistant String (computer science) View (database) File format Electronic mailing list
Mathematics Code View (database) Tableau Right angle Group action output Mereology Template (C++) Window Product (business)
Zoom lens Stapeldatei Functional (mathematics) Structural load Code Set (mathematics) Core dump Bit Simulation Error message Field (computer science) Social class
Web page Context awareness Error message Web page Tableau Group action Wahrscheinlichkeitsfunktion Reading (process)
Point (geometry) Context awareness Functional (mathematics) Group action Feedback File format Group action Mathematics Error message Row (database) Right angle Lambda calculus Row (database) Social class
Point (geometry) Scripting language Execution unit Group action Functional (mathematics) Context awareness Information Real number System administrator Client (computing) Mass Group action Web browser Process (computing) Error message Personal digital assistant Website Row (database) Moving average Process (computing) Object (grammar) Resultant Row (database)
Zoom lens Touchscreen File format Multiplication sign Adaptive behavior Zoom lens 1 (number) Control flow Instance (computer science) Field (computer science) Process (computing) Causality String (computer science) Error message
Plane (geometry) Zoom lens Personal digital assistant View (database) System administrator Electronic mailing list Set (mathematics) Cuboid Fehlererkennung Mereology
all right hello
and uh welcome to my talk i'm going to start sharing my slides here
here we go and this is on my multi submission importer for easy form which was a challenge of a title but i've i've been introduced i'm a net i'm
a developer for six feet up and i work primarily in plum right now but i also do python programming and such from the time being and what really happened here is i had a very specific problem that we were trying to solve
um we had existing forms in the site there were quite long forms and we needed to be able to mass import or submit multiple versions of this and i tried a couple of different things like data grid field view wasn't quite as friendly for our clients and users and the person filling out the form wouldn't be decided necessarily so when we have registration forms and we need to import multiple registrations or multiple instances but have it act like easy form there wasn't quite a ready solution so i did some research and some digging around and started thinking about what would the solution need to entail
and for that solution we wanted to make sure that it was something that the site admins could facilitate they could give it to somebody outside of the site and that they could import mass we want to make sure that it would work with just about any template that we had because the clients were allowed to create their own registration forms so they would be changing this from time to time and it had to be flexible we also want to make sure after working out some of the import steps that the site admins could preview the data make sure nothing needed fixing or adjusting and then once again that it executes all the actions of easy form and that was incredibly important to us so let's take a quick demo of what i
ended up with there we go so right here i've got my
verb conf registration form and if i go into actions i have a new action called import forms from csv that takes me to a page and what i can do is i can download my csv template and i can also import my csv so an example of a downloaded template and i just
downloaded it ahead of time and right here is it just gets all the fields and throws it into a csv so you have all the header rows that you can look against and then i could fill this out and i have an example of one filled out for my bird conference i've got blue sj and big red the cardinal registering for my conference and this is a filled out csv
so i can choose this file import that csv data it's going to fill this in to the save data view which i just borrowed so i could do this easily and more quickly and then once i feel oh this is great i think that also should be big as red cardinal make my adjustments oh this is missing in false no you're not approved no you're not actually approved either and then i can go ahead and import this data and what this is going to do is actually save the data into my save data adapter or whatever actions i have in easy form it'll just process it through those actions and then also um we've got this is an example of kind of the mail my mail hogs working a little weird right now but this is an example of the mail that you would get when that's submitted through so in this case i have mail for uh cardi red me and blue boy because i had a confirmation done so that's the example there of how it works
so what i wanted to go through is kind
of how i got to this point and like i really didn't know that much about easy form and how it actually worked before i got into this so i had to do
some digging and i kind of figured out okay so a new easy form uses dexterity i then kind of realize okay so that works on cc3 form and then clones ec3 form and i kind of dug down and drilled through the pieces to kind of get an idea of what actually supports the forms and the processing and everything and then had to work my way back to figure out how to build this
so first generating the csv and that was actually really easy because and that's just to preview that
page again
uh easy form already had a handy dandy api for getting the schema and getting the fields order and so all i needed to do was point to that and then get it into a csv that could be downloaded so here i have just a snippet of code so i made my download form csv which that that was that page with my two buttons and i made a choice to just use the two buttons because it's one page that my clients go to and it's one page that these users can look for everything so they don't have to remember where to search for things and that rides along with the form and then my button just points to this download form csv when you click on it
and then so what i'm doing is basically just making a call and getting that request i take a date time stamp because i figured i kind of lose track of forms if it doesn't have like this is the day i downloaded this in case they make changes to the form the title will automatically have that date time on it and then i went ahead as well and uh just made sure that i set the header so i know this is a csv put on the file name and then just use the csv module there and the dictwriter to write those field names that i pulled from the schema form and the get fields in order and once again that comes right from the easy form api so that was pretty nice and easy to do as far as getting a csv downloaded
and once again that's just an example in a slightly easier to read format of what an example csv for that form would look like and that way now i have a file that the site admins can send to whoever needs to facilitate the group imports and then they can fill this out
so now we get into actually uploading the import and getting that data back in because that's was the biggest challenge i think was getting it in and then processing is processing it through easy form
so once again same page but this time we have the import csv and the import csv data button right there
and what that button actually hooks up into is my preview csv import view that i have registered here and one thing and i haven't dealt with uploading data as much for forms but this multi-part form data if you've never worked with forms and uploading very important for making sure that file attachment gets into your request
now at this point i knew i didn't want to recreate something from scratch so i was going to try and reuse and recycle as much of code as possible from things that already existed and then just override what i needed to so lots of subclassing and inheritance to try and get this to work
so i looked through once again the code base of what i was working with and i said save data form looked like a pretty good model of what i wanted to import and this was that kind of um that's the save that adapter in easy form so i said why don't i subclass that and just pass my data into that and i noticed that actually inherited from cred form so this gave me an idea of where i could start with this edit form factory that's what i could customize to make the form what i wanted and have it do what i wanted to do
so now i needed to make this intermediate screen so that we could review the import and actually pass the data into this
item here and based on what i learned i knew i could build that structure here so i ended up making my own class here csv import cred and that inheritance to save data form and this view is the view i referred to earlier and in the cct form layout it actually has the wrap form so it just wraps your form in the clone view so it could be in the context in the visual context of a phone site so that was nice and easy to just plop that into that and then i had to override a couple of templates because the native template for save data form of course talks about saved data and i didn't want to confuse my users so i edited this just so i could get the title like review import and some of the text that goes along with that and then this form really what it does is it renders all of the subforms so it renders all of that information and the subform being each form submission or each row in this case also i have the import edit form and that's going to be my custom edit form so i can override certain parts of the form that save data would usually be bringing in
so now i've got spotted what format i need and the biggest thing was what does this need for me to get this fed in how do i need to format my data and i looked and saw get items seem to be what fed the information into the actual slides there so now i have this get items format here and this description i'm like okay now i know how i need to format the stuff that i get out of that csv to start feeding that back into that save that adapter view grid all right
no other way so actually ahead of time just put a couple of code bits together so it's a little easier i hope to display on this but here we go review the import so i talked about my class already and that was my snippet there and so what i'm doing here is i'm importing my csv and getting that data and putting it into this and returning that and then this is actually getting that data so let me see there you go that calls upon this to actually get the data out of the file and so i go through and i actually i run a little cleanup values function because it's user input so i needed to keep control of what the users were putting in and kind of clean that up and make sure it was going to jive what what i got on the end so i had this clean up value here and once i cleaned it up i was returning the format that i needed this data to be in to go into that get items function and this is an example of the cleanup values um and what it really does is just kind of make sure and this is really easy form api really helps me figure that out is make sure that i was getting daytime fields were going back in the way they needed to um the sets in particular so any of the multi-choice text selection bits we're going back in properly and cleaning up all that information so that when it gets read it would like check the boolean boxes it would make sure the drop downs have the right values and it would select the items in the set that needs to be selected
and then that's also took advantage of the cleanup that was already in easy forms api so i actually just use this directly within my cleanup values to make sure that the sets are formatted correctly
so now this is my new get items where i have my i check if it has a file attachment or i check to see if i hit the edit button or if i had to reapply changes at that save data view and then if it doesn't have that and it's missing the file attachment it throws him back to the import forms page so just a little checking in case the file disappears or something happens there
and then next reviewing the import part two which is actually handling the data once it's in there and so
i had my review import table that's my big view and then i needed to make a couple changes to the actual table that was in the form as well just to get the buttons and some of the cues that i needed um in the right spots so that's back to my code window
so there were two classes in particular that i needed to override um the batch class um class two methods functions um batch i needed to override because i needed to export this particular variable here and make sure that it was not trying to read what already existed in save that adapter but reading what i was feeding into it and then i had to edit get data and please excuse my nesting that needs to be flattened out a bit but this was also once again just making sure that i'm iterating through the items setting up my field names and i'm actually checking against the fields that are already in the schema so if in the csv there's something that is in the column that doesn't exist in the schema it's just going to ignore it so it's not going to throw errors it'll only load things that match the titles that are in the schema already so this is doing those checks and then once again making sure that my sets are adjusted correctly so back to this side
and now the biggest thing is the data is uploaded i've got it formatted it's in the table we're ready to hit submit and actually submit this to easy form
and so i chiefly overrid this button handler um and uh this is actually easy forms handle submit so this is how what it does when it submits a page so i kind of read this and observed it and said okay i need to do something like this but in my own context
so this was my entry point so i was already using that import edit form i made that custom form and i was going to override its apply save changes button with my own button and that's my import form data button and i kept a lot of what was already
there but in particular right here this is where i'm starting to iterate through the rows and so for each row in that data is one submission of my form that i wanted to go through so right here i have a function in my import class that's called submit form and so as i go through the rows in the form it's going to process this through that submit form action and submit it as if it was actually in easy form and then just count the rows so i can say these are how many rows were imported just a little feedback for the users on how that's going
and i i made it i made it to what my real chief objective goal was at this point i wanted to submit this form and so i this is my submit form uh submit form function there and i just actually imported the easy form process actions code right there and just push it straight through that process so i didn't have to write anything extra just connect the two pieces and let it roll
so now that was kind of just the beginning it was like my first foray really diving into the forms and just kind of manipulating and pushing things into place to get what i needed to work in that context and it taught me a lot i learned a whole lot and i think the result was pretty cool it was pretty useful for our use case and um of course there's plenty of room for expanding because it'd be nice if they could you know have a clear button or cancel button or add or delete rows and such but i felt it was a pretty nice stub for starting to get through this processing being able to submit forms in mass without having to repeat it one by one so it really accomplishes the goal of passing this information to the users letting them just fill this data out and then giving the site administrators an easy way to facilitate registering all these people while whatever mailer actions or save adapter actions they have or custom scripts it'll just go ahead and process that just mimic that process
all right so thank you for coming along and i guess this is my my time for questions if there's any questions yes thank you very much for the great talk and i really enjoyed it that you showed it in depths how i made this adapters and also and we have indeed uh one question from antony it's what happens during the data cleanup if a value is unusable such as a string that can't be changed to day time etc okay so what i ended up doing is um let me go through here
and uh get back to my cleanup oh the zoom bar is in my way so what i did in this cleanup is um what will happen most times and for date field in particular is it'll just throw the value out um so this is that this was the real importance of having that intermediate screen where if something like this happened they would be able to join the two ends together and kind of say okay well that's not right um or this is kind of off or something like that um so i try and look at it and i try and format it but if it doesn't format it it'll just throw it out so it won't cause an error so that way if things are malformed it won't break the whole process for the users and this was kind of the stub of starting to go through and trying to catch some of those instances and of course with data cleanup it can it can be a lot um depending on how many things you're trying to accommodate but this was based on the forms i had what i could start to accommodate or what i knew i had to be able to accomplish for them i think date time and date were probably the trickiest ones um as far as throwing values go
anything else like um if the sets weren't formatted correctly it just wouldn't highlight the item in the set and for like these boxes here if the value was missing and i already fixed this one it would say missing colon and what the value was so i'll actually just re-import that go back pull that up so it'll tell you that value was missing from that list and then that gives that um administrator a chance to say okay i can fix this based on what they have there so that's what they can do in those cases um there's another question i wonder if this could be a pull request for a new for a new for a new easy form feature so i i think it's definitely possible i think when i started this and this is something that it's really easy um especially if you ever have impostor syndrome i thought this is just a small thing i'm doing in the corner of my world and uh when i presented this to my team they seemed really interested and that's how i ended up giving a talk on this in the first place so i would definitely like to get this out there because i think there's so many great minds in the phone community and we could actually really take advantage of this um so i'm glad to get a stub out and see where it goes from there okay so fred asks um is this something that can be added to the add-on itself i'm not sure exactly what this refers to or what would be needed for doing that can you do you get what yeah i think it's something that could definitely be put into the add-on so when i did it of course because it was in my own little add-on most of this is actually just written most i think all of it is actually in my views file so it's definitely something that now i think about it i would have definitely liked to put this into its own view or own apr give it its own little niche but i think if we could uh create some of the api endpoints or something like that to make this a little easier so it's maybe a little less hacky i think it could definitely be integrated and just be part of easy form i think the cleanup um of the values would be the biggest thing cleanup and error checking okay so there are no more questions on slido