Inside Active Storage: a code review of Rails' new framework
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 88 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/37311 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer | ||
| Production Year | 2018 | |
| Production Place | Pittsburgh |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
RailsConf 201849 / 88
9
14
16
19
20
22
23
26
27
28
34
35
36
37
38
39
41
42
46
47
53
57
60
62
63
64
69
72
80
85
87
00:00
Data storage deviceSoftware frameworkCodeSocial classPerfect groupSoftware frameworkCodeData storage deviceObject (grammar)Level (video gaming)Slide ruleDiagramComputer animation
02:29
Data storage deviceVisual systemMobile appUniform resource locatorElectronic program guideSheaf (mathematics)Data storage deviceLecture/Conference
03:28
Social classLibrary (computing)Data storage deviceLibrary catalogGame controllerPrice indexData modelElectronic visual displayVideoconferencingAngleFiber bundleCoefficient of determinationVideoconferencingGoodness of fitComputer-generated imagerySocial classDegree (graph theory)Computer fileMereologyRoutingComputer-assisted translationSource codeSimilarity (geometry)MetadataResultantElectronic program guideProduct (business)Open sourceForm (programming)Web pageFile viewerBlogData storage deviceLibrary (computing)Parameter (computer programming)Service (economics)Medical imagingWeb applicationCodeField (computer science)Endliche ModelltheorieMobile appGame controllerSemiconductor memoryMultiplication signDigital photographyWeb browserProbability density functionEntire functionAngleInstance (computer science)DatabaseComputer configurationLibrary catalogCartesian coordinate systemOpen setFiber bundleText editorComputer animation
12:10
RootData storage deviceMiniDiscInterface (computing)Social classService (economics)Read-only memoryLocal ringComputer fileLocal ringData storage deviceKey (cryptography)Service (economics)Connectivity (graph theory)Computer fileMiniDiscStreaming mediaDefault (computer science)Configuration spaceCuboidPoint cloudHard disk driveMilitary baseCodeImplementationMedical imagingComputer-assisted translationPattern languageInstance (computer science)Goodness of fitSemiconductor memoryWeb browserMobile appObject (grammar)InformationSocial classInterface (computing)Product (business)System callSoftware developerNeuroinformatik1 (number)CASE <Informatik>Content (media)Error messageUniform resource locatorComputer animationLecture/Conference
16:54
Computer-generated imageryUniqueness quantificationInformationDatabaseMetadataString (computer science)Human migrationTable (information)Coma BerenicesExecution unitBit rateComputer fileEndliche ModelltheorieMobile appMultiplication signDatabaseComputer-assisted translationTable (information)Field (computer science)Type theoryWeb pageSingle-precision floating-point formatData storage deviceKey (cryptography)Medical imagingContent (media)Library (computing)Product (business)Human migrationMereologyNumbering schemeInstallation artMetadataBinary file
19:16
Social classData storage deviceString (computer science)Price indexUniqueness quantificationIntegerDatabasePolymorphism (materials science)Row (database)Module (mathematics)Macro (computer science)Meta elementHuman migrationPattern languageRow (database)CodeLine (geometry)Subject indexingData storage deviceExtension (kinesiology)Cartesian coordinate systemAssociative propertyPolymorphism (materials science)BuildingComputer-assisted translationMobile appSocial classMetaprogrammierungComputer fileStructural loadService (economics)Medical imagingNumberYouTubeInformationEntire functionMereologySingle-precision floating-point format1 (number)Type theoryInstance (computer science)Point (geometry)Module (mathematics)QuicksortMacro (computer science)Equaliser (mathematics)Endliche ModelltheorieTable (information)Computer animation
26:27
Data storage deviceDatabaseDefault (computer science)Scripting languageJava appletModule (mathematics)Directed setCodeSoftware frameworkDatabaseInstance (computer science)Data storage deviceComputer-assisted translationCodeDatabase transactionMultiplication signFlow separationComputer fileUniform resource locatorKey (cryptography)Real-time operating systemRow (database)Server (computing)Web browserMedical imagingSocial classRoutingDataflowCategory of beingService (economics)CASE <Informatik>Commitment schemeInformationLibrary (computing)Process (computing)Message passingSlide ruleOrder (biology)2 (number)Automatic differentiationView (database)MiniDiscRoundness (object)Router (computing)Web pageComputer animation
33:38
Coma BerenicesBlock (periodic table)Data typeXMLComputer animation
Transcript: English(auto-generated)
00:11
Hello, thanks for coming. This is awesome. I see a very full room if you guys have an empty seat next to you Can you raise your hand? Okay guys
00:21
Around if you want to sit down. There's like 20 empty seats Well, thanks for coming to this talk thanks for coming to RailsConf It's been amazing so far a lot of great talks and I had a lot of fun I hope you had a lot of fun as well and
00:40
We are also here to celebrate that Rails 5.2 was released Just a week ago if you have any Contribution to the Rails code base. Can you raise your hand? Can everybody else give a clap to all these? I made it possible
01:03
I'm not gonna go through Everything new in Rails 5.2. I'm just gonna go through one thing, but it's a really exciting and pretty interesting thing and it's called active storage and You know because this is RailsConf, I'm not just gonna tell you what it is
01:21
I'm just gonna go deep inside the code without scaring you too much And so you can also learn how things work and it's a perfect segue to the previous talk so that I can invite you all to become contributors to the Rails code base because you know once you go past the stage of this is just magic and you look at
01:41
The code you might realize that it's Ruby and it's classes and objects and you can also contribute to that as a matter of fact active storage is the newest framework so Possibly might still miss some documentation or maybe you can find some issues So it's the perfect place for everybody to start in my opinion looking at the Rails code base
02:04
My name is Claudio and English is not my first language, so You can just go and download the slides if you want. They're already available there. It might help you follow speakerdank.com slash Claudio B
02:22
so Before I get started I just want to comment that this talk is about active storage Turns out I work for a storage company, but it's not it's not cloud storage It's actually we come to your place, and we you know take the stuff that you don't need We wrap it up take pictures barcode take everything to a warehouse
02:43
And then you can ask for your items back with a Rails app of course and it's called Clutter You can check it out, and we're based in LA and we're hiring and that's my pitch but We're gonna talk about active storage instead Clearly I can't talk about everything
03:02
But there is a great place to start even after this talk and that is the official race guides so at guides the Ruby on res org there's a section about active storage, and it goes through All of it, so it's actually Really well written so I invite you to just go there and by the way all the URLs that you see are
03:22
on my speaker deck so you don't have to write them down and This is what we're gonna talk about today First of all what is active storage? Maybe you've heard about it. Maybe you haven't you're just curious I'm gonna give you an introduction on what it is and how to use it and
03:44
You're gonna see that. It's really really easy to use so then you might wonder why how how does it work? What is it made of and that's gonna be the second part of the talk? I'm just gonna go through the main classes of this library and Finally, how does it all work together?
04:02
And I think you know one of the reasons why I'm giving this talk is because when I looked at the code of active storage It's really it's really elegant. It's really a good code base and sometimes people wonder What can I look at? What is a like a library that is a good starting point? So this is a you know my advice and you can look at that and so
04:24
Enough with the introduction. I think I'm gonna get started Okay, so What is active storage and how to use it? Active storage is a library to upload files
04:41
If you have a web application and you want users to be able to upload files to the browsers This can be a good option for you It's not the only one in the past there have been other third-party libraries like Paperclip or Clearwave But active storage ships inside Rails by default, so you already have it there, so you might want to give it a try
05:03
And so how do you use it? How do you let users upload files? I'm gonna show this in a brand new app so there are no dependencies I'm just gonna create a scaffold brand new scaffold Rails 5.2 app and then have an upload
05:20
so This is how you just create a brand new app in Rails you do your Rails new I call this app catalog and I generate a model called cat of course because it's RailsConf, so I haven't seen too many cats in this Conference yet, so this is my my part for that so every cat has a name and then we create the database we migrate and we run the server and
05:47
If you've ever used the scaffold in Rails You know that it generates forms like this one so you have a form where you can add a new cat to your application So this is the baseline we already have a form now. We want to add a field for people to upload an image
06:04
So there are three steps The first step is actually to just add a field to the HTML, and this is pretty much HTML we are adding a file field. I call this picture. You can give it whatever name you want photo cute little picture
06:26
Cat cute picture and then a label that's that's really it. There's a new field called picture now in Rails whenever we have a field in a form that we want to Submit to a controller we need to tell the controller about this field
06:43
We need to whitelist this new parameter because of the strong parameters in in Rails so what that means in your controller you Already have these params require cat permit name. These are all the prompts that are accepted from the form
07:02
So all we have to do is to add picture as another parameter that the controller is going to accept and then the last step in the model in the cat model we Have to add a single line of code. We just have to say cat has one attached picture
07:24
and That's it. That's literally all you have to do if you do all of this then you Have this form people can upload the picture the picture gets uploaded it gets attached to the cat and as a matter of fact if you
07:40
Have a show page you can just do image tag cats dark picture, and that's going to display the picture right there Do you think it's awesome? Okay, a round of applause for active storage. I think it's pretty awesome Also for the cat So you know this is the basic usage and but it doesn't stop here
08:04
There's a lot more that you can do with active storage Here are some of the other things I'm not going to really talk about here But you can still do imagine you have this cat dog picture You can actually display a variant like a black and white 90 degrees flipped image you and all you need a
08:26
Library like mini magic in your gem file, and it does this it creates another file like that black and white It's not only for pictures I mean for images you can also have like has one attached document like a PDF file
08:40
And there's a matter of fact if you have a PDF You can even display a preview like an image that renders the first page of the PDF and you can resize that 100 by 100 for instance and Of course of course if you have a cat you're gonna have many videos of your cat and so it's not just has one attached you also have has many attached that works in a similar way and
09:06
For each video you can extract metadata the size the angle the duration of the video So there is more about how to use active storage there's variants for viewers and analyzers and so read the guides and
09:24
Also, if you just google active storage how to use active storage Probably the first result is going to be a blog post by Vladimir who's right here Alex by evil Martians, and they have this very
09:41
Comprehensive blog post about how to set up all these features, so Really, you know if you want to start using it. It's pretty easy to start and if you want to do all these variants and reviewers You know please feel free to do this just try in a brand new app, and then see if it works in your production up
10:06
Okay, so far so good because this was the easy part Okay cool so now How does it work like what's inside the code you know like? You know what we were talking about in the previous talk. You know you use the routes
10:24
But then you want to see how they work so you open the the code base so here we're gonna go through a similar journey and We're gonna look at the classes that are inside active storage You Well to look at the source code, how do you do that?
10:44
Rails is open source so the entire source code is on github so you can just go to github.com Slash rails slash rails there is an active storage folder and the entire code is right there Another way to open source code if you have active storage library in your app
11:04
You can just do bundle open active storage, and it's gonna open the source code in your editor You can actually do this with any any library. That's in your app any gem Okay, so you open the source code, and there's a bunch of files. Where do you even start so this is?
11:25
This is what you should remember about the important classes of active storage There are three Main classes there's active storage service active storage blob and active search attachment And I'm gonna explain what they are so next time you hear about a blob. You know what exactly we're talking about
11:51
So let's start with active storage service The service is the part that deals with Just moving bytes. You know you have an attachment you have a file
12:02
Just moving the bytes like from memory from your browser to disk. That's all the service does so I tried to Make a picture of this when you upload the cat picture From the browser so an HTTP uploaded file
12:22
There is a component that takes those bytes and stores them for instance on your local hard disk if you're in development in a certain folder Storage so that's the service. That's what the service does it moves bytes from memory to disk Active storage service is an a real class and the code looks like this
12:42
it has other methods has upload download delete and so on these are the most important ones and Funnily enough this is the implementation, so if you call active service upload it raises an error This is actually a pattern what this is is an Interface this is basically saying there is not just one service
13:04
There are many subclasses, and you can implement whatever service you want But they have to follow this pattern so you can't call active storage service directly you can call one of its subclasses and not Subclasses they implement an upload method a download method because they can be implemented a different way
13:24
So this is just a pattern you might have seen even in other Ruby code bases that there is a class It doesn't actually implement the method it just describes. What what methods are there? But now let's look at one of the subclasses
13:40
This is called disk service, and it's the one that by default so in the examples I showed before I was running an app locally and disk service. It's the default configuration So this service the upload file the upload method it takes this IO this file And really all it does it's calling the Ruby
14:02
Methods IO copy stream that just copy copies bytes because this is the one service that stores a file on your own Disk so it's doing anything else that taking the bytes and taking the stream of bytes and copying to a certain location The location is this make path for key which by default is inside this
14:24
storage folder in the same revs app The download method is also using Ruby Libraries, so it's a it's reading the content of the file from the same path so this is a pretty straightforward implementation takes the bytes move them from
14:43
memory to disk and the other way around So this is great for development It's going to store the image on your computer But for production you probably don't want to store the image locally you want to store it on a cloud solution like for instance Amazon AWS S3
15:00
And the good thing about active storage is that it already ships with a service for S3 So you don't have to build it. This is another class. It's the S3 service You see that there is still an upload method and a download method They're just implemented differently This object for key put those are all methods that come from a gem called
15:23
AWS SDK S3 gem that it's included So it's just using the methods to move bytes to and from S3. So if you want to use active storage with S3, all you have to do is You go to the configuration file. It's called storage.yaml
15:44
By default, it's gonna say service disk. That's the default You just change that with service S3 and then make sure you have the credentials for a bucket on S3 And then it just works out of the box and so this is pretty convenient and
16:01
active search actually also has support for Microsoft Azure and Google cloud storage on inside and if you have your own cloud solution and you want to build a Service for that, you know, just make sure you follow this this pattern this interface and then you can build it And maybe you can build your own gem or maybe you can do a pull request to integrate it in the code base
16:26
How are we doing so far? Thank you. Well, you can applaud if you want Okay, so service moves bytes so in the case of the cat picture we have the bytes
16:42
But we don't know what those bytes are. We have no idea that it's even an image It's just bytes. So we need some place to store the information Where is that thing what it is? And this is what the blob does Same example when you upload this file The blob is the part that it's actually storing the key
17:01
So, you know what the file is the original file name might be useful the content type. It's the JPEG and the byte size and So this is the part that stores a reference to the file and It's actually storing it in the database active storage blobs. It is a table in the database So this is the scheme of the table and it reflects what I was just describing before. He has a key a file name
17:26
Content type metadata byte size and the checksum to ensure that the image is not corrupted Now the you know, you might be wondering. Okay, where's that? Where does this table come from?
17:41
We did not create any table so this is one of the you know things that rails helps you do because The first time you ever use active storage in your app and you try to upload the file You will see this page and it says could not find table active storage blobs
18:01
So active storage needs that table to work But also it tells you what to do to resolve this issue run bin rails active search install And this was actually my own small personal contribution to active search But this is really the only thing you have to do only once so that you need these tables
18:21
The first time you try locally you might see this page then you run this command This command adds the migrations to your app you run the migrations and then the tables are there and then you don't need to do Anymore when you deploy to production the migrations are gonna be there you run them and that's it So only once you need to make sure that your app has these tables and then that's it and
18:45
Just as an approach. This is a little different from other File upload libraries that instead require you to add fields to every single model So let's say your cat has picture they say well you need to add a field called picture to the cat Then you have a dog model. It says you need to add it there
19:03
Active search is a little different because everything is is here is inside these tables So you only have to do that one. So I think that's pretty convenient So I'm just gonna show you one methods of active search blob and this is the upload one. So what it's doing
19:22
as I'm saying it's as I said before it's Extracting all this information so it calculates the checksum from from the bytes and then it extracts the content type either from the Extension or the mime type or just the first bytes and then it extracts the size it stores all this information And then it calls the service to actually store the file and then here's the last class
19:49
So we have the bytes we have the blob that it's telling us. This is an image of this certain size But we're still missing a part. We are missing the fact that this image is the picture of this cat
20:01
We're missing the association. So that's the last part that it's missing to associate The blob with what it belongs to and this is what active search attachment does So this is the you know The last part when you upload a picture of a cat apart from the service and the blob There is going to be a new attachment that says this is a picture it belongs to the record
20:25
Cat number four for instance, and this is the blob number two. So it's really just building this association So then when you call cat dot picture, it knows where to go and active storage attachments is also a table and it's the other table that the migrations add to your app and
20:44
This is the entire schema. So it has a name the polymorphic record and then a blob and this index So what this index is saying is if you have like cat number four It can only have you know one picture for this blob
21:00
and that's that's Basically all there is even the code for attachment. It's really just describing, you know what I just Mentioned it's backed by this table. It belongs to a record polymorphic So maybe the cat and then it belongs to the blob it creates the association
21:22
And so summing up these are the main classes you have the service that moves the bytes you have the blob that has a reference to Where the bytes were stored so then you can fetch that and then you have the attachment that connects this blob to the original model
21:42
Okay, so this is like the third act where we like put everything together How does it all fit together at the beginning? I show you Active storage is pretty easy to use it. Just add a few lines of code Then I mentioned the main classes, but how does it all work together? There is still some magic there that it's not pretty clear. And so that's what I'm going to show you now and
22:05
This code uses some techniques that are actually all over the rails code base And so I know if you learn some of these things that they might even be useful if you look at other parts of Rails or even your own code base
22:21
so I Think the perfect starting point is when I said, well, all you need is to add this line of code So how does this single line of code add this whole behavior to your cat? it's pretty powerful and Even before that where does this method even come from because here we are inside an active record
22:41
base active record doesn't have has one attached as a method so How did this method end up there? Well, the answer is that active storage is an engine and James Adam yesterday had a talk called here's to the crazy ones where he talked about engines for
23:03
The entire talk so it's a really good talk to Watch on YouTube you have the chance Active surge is an engine. So as an engine it has Initializers and this is how this code works. This is inside active storage
23:20
but this is saying is Whenever a race app is loaded is in is initialized Active support on load active record what that line is saying is whenever you're loading a race app when you're loading active record stop there for a second and Extend active record with active storage attached macros
23:42
So it's loading active record with all the active record methods where find and so on and then once it's loaded extended with active storage attached macros which is a module inside active storage and One of the methods is has one attached And so that's where that method comes from that's how rails is able to
24:05
extend In in this case, we're extended active record with active storage, but this pattern is really used all over Okay, so that's how the method ended up there now. What is this method? This matter is actually pretty long and I didn't put the whole method here because it can be scary
24:25
It just stopped at the first like five lines, but even just the first line is like class eval code What is this? Maybe you don't write this type of code normally in your application so Let me explain what this is When you say a cat has one attached picture
24:43
Suddenly your cat has a method called picture This is pretty similar to when you say in active record user has many roles and then you have user dot roles So you have this new method just because you called something else. Of course rails cannot know at the very beginning
25:00
Has a cat has one attached what it has to take that name that you're giving it and then Define methods with that name. So this is what's happening here This class eval is taking that name that you pass that has one attached and it's using that to define a bunch of methods with that name For instance if name is picture this code becomes a little easier to understand because you said
25:25
Picture now you have a method called picture equal and then there are medical picture and so on So this is really sort of meta programming in rails, but and it also helps you when you debug to go to the Correct line, but this is really what it's doing. It's defining new methods based on the name that you've given
25:46
now this picture equal is the method that it's called when you Set a picture. So when you attach a picture to a cat and it's calling picture dot attach Picture itself is another method defined by has one attached
26:02
So this is the picture equal This is the picture method Picture is an instance of an internal class called attached one It's it's this is what picture is the good thing about this is, you know You can go and look at the code of attached one, but it just gives you an idea of how really
26:23
object oriented This active search is it's not like injecting I don't know loading modules It's really based on a bunch of classes and these classes are pretty small So if you go inside you're gonna see that they're not too hard to understand so this is an instance of this attached one and this is the
26:46
Class that has a method called attach In this method, it's basically doing two things First it's creating a blob from your attachment and the blob then calls the service to move the bytes And then it's creating the active server attachment
27:04
So we go back to where we were before because you are attaching a picture to a cat first it's creating a blob and then it's You know storing the bytes with the key and then screen the attachment So it goes back together to where we were and this is
27:21
you know, I hope they're like with this talk and really like letting you know that you can just go and look at the code base and Both learn and you know possibly contribute as I was saying before this Library is is pretty new and it's open to
27:40
You know suggestions. Maybe there are some issues Something else you can do if you go to the github page of rails. There is the issues tab All the issues are normally tagged. So if you see the tag active search you might you know want to explore and see What happens in the code the code base is not really that big
28:02
You know, we heard about journey before that's a little bigger as a code base But if you especially if you're new to rails I think active search is a good place to start active support is another good place to start active job as well They're all pretty small. So just give it a try. I Just want to give it another like round to explain again all this
28:24
process of Attachments blobs and services This is another method that Active storage ads we have has one attached picture. It's this after destroy commit so think about it if you you know destroy a cat which you should never do but if you
28:46
If you decide to destroy it, this is our cat from your database and that cat had a picture You probably want to delete that picture So if you had a picture on s3, it doesn't need to be there anymore and maybe it shouldn't you know a cat picture Maybe you can stay there. But if it's an attachment of a document and you delete it
29:02
You don't want the attachment to stay there So this is also something that active search implements with this after destroy commit picture purge later After destroy commit is a callback of active record and it's saying do something after The instance was destroyed
29:20
It's it's after the strike commit so it doesn't run inside the transaction of the Deletion if it would just have to destroy it would it might be blocking your database So, let's see what happens when you destroy an instance of a cat. It's calling picture dot purge later Purge later, it's saying if there is an attachment called attachment dot purge later
29:44
Now attachment you remember was in the database. It's destroyed but then it's also calling blob purge later The blob is actually calling a job is saying well, I don't really need to delete this file in real time I mean I can just let an active job do this like five seconds from now
30:02
When the job runs it's calling blob purge Blob purge it's it's destroying the blob and it's calling delete Delete is calling the service and say can you actually delete this file and then the service for instance the disk service It's calling file dot delete And so this is really the entire flow and you know, I just wanted to show this again just to you know
30:22
Demonstrate some some properties of actually search for instance in this case really separation of concerns Like the cat doesn't know that the file is even on s3 just going step by step. So If I haven't said yeah, I think this lever is really awesome and the code is really elegant. So check it out
30:41
To conclude there is more and I didn't have time to go through that but Some of the other things that you may want to explore First of all active storage is not just a Ruby gem. It's also a JavaScript library The code is is in the same place And the reason why there is a JavaScript library is because sometimes you don't want to upload your files
31:03
To the rails app you just want to upload them directly from your browser to s3 for instance especially maybe if you use Heroku, you don't want to use the storage of her. You just want to upload them to s3 so active storage includes a JavaScript library to do that you
31:22
include the JavaScript file in your in your view and it takes care of Creating a blob so you have a key then with this key The file is uploaded to s3 and then this is the key that You can then use to retrieve this image The only thing that changes is the order in which the blob is created and the image is stored
31:44
Something else that I found, you know useful is Every time you use active storage in your rails log You're gonna see that information like file was uploaded and so on the way in which This is done is using a library called active support notification That you can publish a message to and then subscribe
32:01
It's used all over rails. So, you know, if you find this and you don't know what it is, just check it out It's pretty good and then the last thing is Active storage has a file called routes letter B and I like that we tie back in with the previous talk about routes
32:20
It has routes because imagine if you have a file then you want to display that file in the browser So it needs a URL. So this URL for a file needs to be generated The nice thing about this file is that about routes that are B Is that it's using a couple of methods in the router that were just added in rails 5.1 the methods direct and resolve so in your router you can have
32:45
You know resource get post but you can also use these two methods and maybe you have never seen these methods Used in the real app, so if you want to check them out they are in active storage and This concludes my talk all the slides are available there and
33:05
Really? I don't give it a try and let us know what you think I am a member of the rails issues team, which means if you open an issue in rails, I can you know look at it and check it and maybe merge or comment and
33:20
Even if you want to mention me on GitHub, you know, I am clouded beyond get up so you can do that if you open a PR it will make me happy And that's it. I'm gonna be here for the rest of risk up. So if you have any questions, just come find me