Open Sourcing: Real Talk

Video in TIB AV-Portal: Open Sourcing: Real Talk

Formal Metadata

Open Sourcing: Real Talk
Title of Series
Part Number
Number of Parts
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
Hired open-sources some useful abstractions from our Majestic Monolith® and we've learned a lot. Some tough lessons, and some cool knowledge. We'll cover: When & where should you pull something out of the code? Does it really help? What things are important to think about? What if it never takes off? We'll also look at some design patterns from our open-source work.
Covering space Theory of relativity Open source Multiplication sign Ultraviolet photoelectron spectroscopy Real-time operating system Parameter (computer programming) Open set Twitter Goodness of fit Computer animation Different (Kate Ryan album) Mixed reality Blog Quicksort Writing
Dynamical system Server (computing) Statistics Open source Information Multiplication sign Projective plane 1 (number) Bit Graph coloring Web 2.0 Uniform boundedness principle Process (computing) Computer animation Bit rate Profil (magazine) Query language Software framework Data conversion Figurate number Middleware
Trail Mobile app Randomization Open source Wage labour Code Multiplication sign System administrator Range (statistics) Online help Student's t-test Mereology Software bug Twitter Number Average Software framework Dialect Copyright infringement Software developer Bit Total S.A. Cartesian coordinate system Wave Computer animation Logic Personal digital assistant Freeware Thomas Bayes
Open source Staff (military)
Axiom of choice Open source Code Decision theory Mereology Disk read-and-write head Web 2.0 Different (Kate Ryan album) Profil (magazine) Endliche Modelltheorie Social class Physical system Graph (mathematics) Bit Cartesian coordinate system System call Ruby on Rails Arithmetic mean Process (computing) Computer animation Pattern language Cycle (graph theory) Figurate number Library (computing) Directed graph
Trail Functional (mathematics) Group action Open source Code System administrator Multiplication sign Water vapor Parameter (computer programming) Event horizon Mathematics Endliche Modelltheorie Physical system Condition number Social class Validity (statistics) Block (periodic table) Weight Interface (computing) Projective plane Analytic set Plastikkarte Bit Database Unit testing Cartesian coordinate system Grass (card game) Wallpaper group Computer animation Personal digital assistant Logic Website Right angle Object (grammar) Figurate number Window Row (database)
Group action Building Functional (mathematics) Context awareness Open source Link (knot theory) Code Patch (Unix) Sheaf (mathematics) Water vapor Mereology Event horizon Latent heat Mathematics Lecture/Conference Software design pattern Core dump Software testing Selectivity (electronic) Extension (kinesiology) Hydraulic jump Social class Information Mapping Software developer Cellular automaton Bit Database Instance (computer science) Cartesian coordinate system Message passing Process (computing) Friction Query language Repository (publishing) Chain Statement (computer science) Pattern language Pressure Library (computing)
Message passing Computer animation Query language Code Bus (computing) Elasticity (physics) Pattern language
Filter <Stochastik> Functional (mathematics) Multiplication sign Set (mathematics) Water vapor Parameter (computer programming) Event horizon Bookmark (World Wide Web) Revision control Latent heat Different (Kate Ryan album) Elasticity (physics) Endliche Modelltheorie Physical system Social class Oracle Algorithm Dependent and independent variables Multiplication Matching (graph theory) Relational database Interface (computing) Database Instance (computer science) Grass (card game) System call Microprocessor Subject indexing Type theory Data management Message passing Computer animation Query language Software repository Network topology Blog Order (biology) Chain Statement (computer science) Pattern language Object (grammar) Abstraction Resultant Row (database)
Filter <Stochastik> Multiplication Computer animation Abstraction
Point (geometry) Complex (psychology) Mobile app Open source Software developer Multiplication sign Bit Cartesian coordinate system Mereology Computer animation Query language Logic Personal digital assistant String (computer science) Elasticity (physics) Pattern language Endliche Modelltheorie Row (database) Library (computing)
Point (geometry) MIDI Building Greatest element Graph (mathematics) Block (periodic table) Code Multiplication sign Line (geometry) Process (computing) Computer animation Rewriting Energy level Data structure
Complex (psychology) Mobile app Open source Code Interface (computing) Flow separation Field (computer science) Word Goodness of fit Computer animation Logic Query language Core dump Elasticity (physics) Software framework Pattern language Endliche Modelltheorie
Point (geometry) Filter <Stochastik> Wage labour Open source Code Multiplication sign Insertion loss Water vapor Mereology Field (computer science) Twitter 2 (number) Neuroinformatik Revision control Green's function Physical system Projective plane Bit Cartesian coordinate system Flow separation Ruby on Rails Tablet computer Vector space Friction Query language Network topology Self-organization Website Pattern language Quicksort Resultant
the and you might if we don't cover here's and go ahead and get started this is open source in real time good as to whom I am inter
evidence I work for hired have been there a little over 2 years have worked for all kinds of different start ups for like 9 10 years and they're working with Ruby on Rails and since about about 1 . 8 or so have anybody remember some fun Twitter related arguments back then good times and in open writing blogs in PHP year Cubase Edgar all sorts of things that a for longer than I care to remember as this hired hired
is the best way to get a tech job as a candidate you make a profile we which alive in our two-sided marketplace and then companies you had to interview with them and they have to give you salary equity everything else up front and you decide if it's worth responding in having that conversation has company this is the best and fastest way to get great candidates and your pipeline however this is not a pitch talk so I'm not going to focus on that 7 a talk about
open source this is something that it's been doing and basically since the beginning we have 46 public repose on get all of those are our forks where we needed to make some modifications or thought about modifying things have at some of those are open source projects that maybe you can use and these are just a couple we have a a pub sub layer over psychic we have a query builder for search we have fortitude which is our uh CSS framework which is built to scale that's built for teens and we even have some smaller ones we have direct middleware that will log stats from the true of web server so if you're using that it can be helpful for gathering more info and we have a tiny thing have which will change the color of the lights in your office if you're using something like Phillips you we turn our lights red when the master build fails has that's a good incentive to uh get on slack and figure out what happened I'm just going to dive deep into these 2 so we can talk a little bit about what producing these projects were like I'm going to cover whether it would be worthwhile or not for you to do any open-source work extracting anything at out and making it public minutes talk about some of the challenges that we faced in doing that in a minute talk about how it played out across our team might have that affect the dynamics in a minute try get a little technical and talk about what we learned about Ruby and Ruby gems and rates and hopefully will have some time for tuning at the end that's why should you
open source things and well if you've got some logic in your application then clearly every other Rails app is doing pretty much the same thing as if we all share the code then we could all do this stuff automatically I new open source something the community will fix bugs and people add features still documented thinks it's free labor it's amazing I get developer street cred you know you'll get lots of respect and appreciation will get 10 K followers on Twitter and hundreds of thousands of get help stars and people will open issues and say well there's no bugs the problems with this I just wanna say thank you and we love you so literally none of that is true that you may have its application code that everyone needs an across basically every real out in which case what's it like working part but and but for the most part you may be doing some pretty specialized stuff most of Europe this business logic so that lives in your business at uh so we'll talk
about our projects in articular I pulled some data off of best gems have bastions basically scrapes data religion stockwork every day and tracks like total number of downloads average number of downloads etc and it gives a general rank out of the 130 thousand + jams on regions I can see are not quite in the top 100 or 1 thousand uh were probably not going to be on the public leaderboard so many times in our average daily downloads adjusted and got rid of a house weekends in like holidays for that have basically it's a little bit over the range we'd expect for how many times a day we deployed parochial uh so hopefully these are all just you know get push a rope you all look we got were downloads and I also holds in data off get half of the number of stars and followers we have is a bit more encouraging and there's probably some people outside of company they care about these because and by you a we're not getting people opening issues just saying how great we are I've found some gems that are similar on the bastions ranking as 1 of them is a framework that lets you build parts for Google Wave as anybody remember that that existed for a hot 2nd out 1 of them is a command line interface to the Pirate Bay I think the founders announced a few months ago that they've been student thrown in jail too many times so that step out 1 of them combines active admin trailblazer a trailblazer is the object-oriented explicit over implicit uh loosely coupled well organized framework that you can add on top of your rails out active admin is literally the opposite of every 1 of those values so I'm not sure what you get by combining them and the last 1 is the command line at the gives you a random Mitch Hedberg quote that I used to use that all the time I mean I still do but I used to to yeah uh
so what we found is if you don't market your genome than people don't really just find it organically we haven't done a lot of promotion for these so don't post about the monarch walk too often this may be the 1st talk where we've going deep into our open-source staff at rails come it
and so we haven't gotten a lot of people that just found this out of the blue and said this is the greatest thing was still worth it for us
to open source this code from our application and I'll died in a little bit and we'll see that's a reactor is our enhanced interface decide and this is how we do a lot of our background processing and hired and so I think if you don't know is the your background processing library for Ruby on Rails application is made by my per him that they have Mike is also thank you very much it's been wonderful system for us higher and it's been rock-solid and so the thing that we were looking at when considering reactor was when you have say a candidate updating the profile you may want to do a couple of different things in the background you might want to notify the talent advocate that's working with them you might wanna bust a catcher update a saved searches something like that and so it's I think you have worker classes for these things and you performing sink and those things will happen in the background outside your web request uh you're doing a whole bunch of things then you may have a whole bunch of different workers that you have to tell to do the thing found so you face a choice do you want to copy and paste all of those calls everywhere that year updating a candidates profile and or you can have 1 worker that fires off and then that calls other workers which might then all other workers have for the most part this is a huge deal I think it's a recommended practice but if you like mean you tend to forget things then you might through a cycle and here workers and will stand out 110 thousand jobs however many I activated until you can shut down sidekick and figure out what happened and so
basically with that pattern you end up with this big graph of things are happening throughout sidekick you have to kind of model that in your head and it can make it a little difficult to track down where exactly specific jobs are coming from so reactor
is a publish-and-subscribe system basically you publish an event that has a name which is just a simple you can pass it any arbitrary data and there's also an extension so that you can have active Record objects publishing event and then those objects are serialized in the serialized and available within new event code yeah and so this will kick off a single psychic worker which then locates all the subscribers to that particular event name and it will in Q for each of those I we wrote a convenience method for our controllers called action event and this whole publishing event from the current user and it will merge in parameters there any other data that we think would be useful for tracing tracking analytics etc. so the subscribers look like you include this ascribable model that gives you the class method on event and so you say on events you give it the event name and then you give it a block of what to do in that event happens and so in our 1st example we plot the actor which is whatever published the events we get the right the and then we can bust a cash related to that particular record have you can also set a wild card in the water so this thing will respond to you all events and we use that for logging site activity to pose grass or to anywhere else that we want to and have that analytics data available there's more options to it instead of passing of blocking compositive event name if you're objects and functionality or a little more complicated than you 1 unit test and this can make it a little bit easier you can also pass options to it like uh delays and uh performed weight so for example once an interview request happens of from this subscriber then 2 days later will send an update and you can use of logic in that like figure out if it's canceled in the water we also have a system for creating model-based subscribers so in this case we create a subscriber called that notifier this is going to live in that are database and we define the on fire code to say what happens when this event fires and the neat thing about this is you can have it subscribe to multiple different events from the database so if I'm an admin and I want to be notified when these 3 different things happen then I can just do that for admin interface and not have to deploy new code to handle that and this can be a little bit magical so how we their use has fallen off a little bit and finally you can have records publish events that based on the data in those records so for example if we have an interview that has a start at an end at time then we can tell it to publish events at those times and we can even pass it has conditions and some other things similar to uh validations in callbacks and the nice thing is if the model gets updated the starter and that changes that will automatically reschedule that for us so did help that we took this code and instead of just building in in our application we put it in an open-source repel well when you do
that and you get this nice greenfield project in actually changes your desktop wallpaper to the old windows XP wallpaper have Greenfields great you don't have to worry about the legacy code I can just start developing fresh and it keeps your
code as separate the Kansas out of your application and so that this is a
little bit of friction to updating the library which turns out to be a really good thing the code for reactor is kind of out of sight out of mind for day-to-day future development for us and so it discourages making small arbitrary changes the founder like 0 this should be called an instance methods not past methods for this is that I'm less likely to take that on because they have to context selection and published Ruby gems and I go through this slightly higher friction process and because it turns it into a lower churn section of code it keeps the pattern in convention clear this action event is used all over our application and basically it stays the same everywhere and it has just kept working and we had to make some updates for the rest 5 then I'm about to make some updates for the psychic 5 but for the most part we don't have to think about it too much also when you have this in a public repository instead of your private Application repository that's going on your permanent record on get out of the cell that might add just a little bit more pressure to do it right you might be a little more attentive to things like documenting all of your methods and making sure that your test coverage is 100 % and that your code climate score isn't going down and finally when you're publishing through RubyGems specifically after be a little more thoughtful about the versioning how when you're making changes you really have to think about OK is this going to break anything is this backwards incompatible process just that feature or a patch released and that isn't going to break anything existing but maybe provide a new feature also when you're building outside the application and you're not thinking so much about features gives you a chance to think more clearly about the pattern of what you're building so instead of thinking about what I need to do this feature you think about what is at the minimum information that this open source library needs to do its job so you also have to think about how Ruby gems in rails Bundler and use other tools work and 1 thing I stumbled onto you pretty early is that requiring rails it's harder than it sounds at hand I looked through things like devise and paper clips and psychic and they all require rails in totally different ways in which I find amusing and some of these libraries were require rails for all of the functionality and some of them have a few rail specific extensions and back Osmond diver little deeper into OK will what is frailty is that something I need and where do I put the requires statements have manage the dependencies for that and and then it makes you think about whether you want to use real specific extensions inside your Janice living outside your application so do we wanna use uh active supports concerns pattern which is pretty nice way of extending objects or do we wanna use that basic Ruby uh include extended patents and then you think a little bit more about like the overall design pattern so when we're thinking about reactor rethinking I just want this action event method in my cos I can just put events all over the place and when you're designing a jump from the ground up it's like all of it was the name for that pattern and so when you go through you know design pattern wall posted the Gang of Four book or whatever you say 0 this is a message by us so we're going to make channels that events get passed along these events are going have arbitrary data an some people get this confused with event delegation particularly if you've done a lot of J query you can have multiple listeners for a particular events there and but those actually form a chain and any of the listeners in that chain can aboard the entire process and with the rest of us all of the subscribers get the message regardless of what each of the reviewing individually that so if you wanted to see some examples of subscribers in message busses written rails the Chinese New action cable has pretty well put together well documented example of this and have a link here to the subscriber map and or if you want something a that water core active support notifications has been in rails for quite a while and if you're wondering how all of your database statements get from the application code your development Council this is what's doing in of this has a link to subscribers in the class is responsible for organizing and mapping them so till the
R publishing the reactor Jan I
did make its famous can really help establish any of the selfish goals that we put out and but it did help a team established a convention it has quietly kept working for years and it's allowed us to write their code in a way that smaller and more loosely coupled and better organized it's also a good excuse when anyone wants to work on it to go in and look at the pattern with the examples and see how does message bus work what's cool about this the other 1 I wanted to talk about was stretching this is a composable query builder for elastic Search and elastic
search is a pretty amazing technology for full-text search as based on we've seen it similar the solar and basically when you want something more advanced than a relational database I like statement then this is a good place to go and yes we spend an
inordinate amount of time thinking about the name for this uh because no GM is going to take off and become popular unless you have a really clever upon is the name for it and so I wrote the 1st version of this the goals that I was thinking of of course water I wanted to have an active record each query syntax that we can use for elastic search in order to have the it's immutable query objects which make handling and caching responses really easy to manage and I want this to be just a query builder have for indexing our models to Elastic Search and we already have reactor and so we can index on specific events which serializes so I think it would just keep keeper ElasticSearch jam out of all of that business I want to make is flexible enough for pretty complicated elastic search queries and because this is just a query builder it's taking a bunch of method calls and then it's producing some J. song that we're going to send Elastic Search and that means we don't really need any dependencies uh which is pretty nice we don't have to require all frills for this 1 thing and and the end goal of all that is we have these composable scopes uh that we can use it in the same way that were familiar with from Active Record so here's an example of what that looks like and this is a query that should get us back Lynn Conway as she basically invented modern microprocessors so that's a good person defined in a candidate database and you can see here we have the where method pretty similar Active record you can throw some fairly arbitrary parameters added and it will figure out what the right query is to apply those and but then there's some more elastic search specific stuff this match query for example in is like the most basic full text matching relevance sorting query inelastic search uh and then we have these boost methods have that that gets into what's called a function score query lets each week at how or your search results are going to be ranked so when you look at the query that's generated it's this giant message a something and when I 1st started working with Elastic Search I tried to figure out how to make these giant Jason some documents are composable so that we can mix and match them in different ways turned out to be really hard which is what inspired this whole Active Record like approach so some other example methods altitude you can but you tell but what index you're going to search for a type you wanna get back at and then you can use where forward filters and you can use match for the full text matching and you can of or multiple filters together and by using the should clause this comes from straight from the elastic Search had DSL and then each at boosted tweak you relevance algorithms and finally if what stretchy has isn't enough then you can go lower level you can say this is just a query and throws rock chase on and that it will figure out where that needs to go in your final query that's going to be sent over uh ElasticSearch has a whole lot of stuff in it had has about uh 7 million tools for uh doing different kinds of searches and figuring out relevance and producing this set of results that you want and so it became obvious pretty early we can support all of that and so the goal was just to get to a system where we could do all you needed to AP and then hopefully expand that as we got more and more features in hired and so how do we get started making an Active record Chernobyl interface and I pulled out this 2010 blog posts that had bookmarks forever uh real stitched or thinks John Nunamaker this stuff so works that's still pretty much the same in Ruby 2 3 and basically you have a class that class has data may be in a array of nodes let's say and each of your chain of all methods returns a new instance of that class and with some new stuff added to it so when you get down you like OK were building up this gray and then we're going to compile it is Jason query and what is that look like this and mean for that has worked and turns out it's called the visitor pattern and I was looking around and how other repos do this and I wanted active record and so that users their L and Errol has the visitor pattern so that has a collector class and then it has a tree manager for figuring out what was collected in compiling it and then it has multiple different visitor classes to take that tree of abstract nodes can compile it into a query for a same mice full of post grass or oracle I guess if you're into that kind of thing and this is an
example of what the Treaty of abstract nodes is gonna look like and you have to select a node and then that can have multiple child nodes like are you selecting where you getting it from and then any filters that are going to be applied to your query and and those filters can have child nodes and those have child that's the
so the visitor will traverse
all of those and compile it into the final string so stretching didn't need
that full has stack there's really only 1 Elastic Search we decided because time now we only wanted to support the latest version and so we can basically combine the visiting and compiles steps the pattern was still pretty much the same as some challenges we ran into in having this query builder live outside of our repelling kind of its own thing I was a little confusing to the rest of the team initially if you look at other gems for integrating rat ElasticSearch interior Rails app and they do a lot more ElasticSearch rails and search kick in and Julie and they all tied into you active model they have some other miscellaneous helpers and stretch it is a new idea that so that other developers will account write do this without stretching and it's like well we don't actually do that and it turns out when you have this API like Active Record the whole point is to make it easier so you don't have to look at all the documentation for everything all the time but that turns out it hides a lot of the actual features of ElasticSearch hide the underlying complexity so other engineers will look at this and say I'm just gonna make this active record like thing but it's like maybe you're missing out on a lot of stuff you could be doing there and when we have an open source plus internal application queries can be a little fuzzy deciding what should go in the open source versus what should go in the AP and obviously all of our business logic should be me have but if we're using this particular style of query for text relevance for that particular style like is that worth having a convenience method for in the open-source package yeah and then early on we had a couple of cases where it's like OK we wanna make this feature and that then the gem doesn't support that yet and so you're open-source library ideally should not be walking in the future development so had to fix that part at least that we had to make it more flexible and easier to customize for their application and then a little bit later we made of more flexible and extensible so that we can optimize queries of the application and then later we may be easier to customize the hand more flexible and more extensible and
that turned out to be a huge win actually you can really see that process of editing and refining get flexible extensible solution and because ultimately it means less code and uh it makes clear what the bottom level versus mid level building blocks are as well actually graph this
for the structure the and it's urged each of our a master commits and looked at how many lines of code a in and so with done 1 2 rewrites as the first one and mechanical bad but it certainly at a lot of code and then over time we've been able to just keep cutting back down and at this point it's back to being a pretty small and
so the till the after that
and by making this open source genome in this nice field have we got to learn the cool new visitor pattern and see how do you do that in Ruby have for me it was a great excuse to dive in and read a bunch of code from their L and Mongoloid and other like more mature frameworks and I hate having the core interface the open source that allowed us to keep that very simple and then keep our complex business logic in the app word belongs and it's helped us keep a good separation between elastic search queries and the internals of like our models and are controllers so to wrap up the whole
thing open-sourcing has been pretty cool for us and even if you know it hasn't made us rock stars we don't get you know a Lear jet to fly riddles Conf because of how loss we are I even if this guy's yearly water on get home and you can still get a lot of technical benefits for your organization and even if you don't afree labor it can help you have internally with your team I can
add a little bit of friction which can be a good thing I can be a bad thing occasionally but we found the trade-offs to be worth at most of the time uh thinking outside of Ruby on Rails and sort of leaving behind the conventions in crèches seconds you act major simpler and clearer and easier to understand and developing open source that can give you a good excuse to learn more about computers to learn more about Ruby and Ruby gems and underwear and all the other systems for the how you get that back in your application and with a green field blank slate projects projects it's a great opportunity to be a little bit more thoughtful about the code that you're writing so I haven't told you which save time for tuning so here we are uh any questions for me about these open source projects or about hired yeah so the question was uh ElasticSearch uh change versions from 2 point X 2 5 because reasons and they completely changed how the query uh DSL works there used to be this big separation between queries which can handle relevance and then filters which shows a vector result set that don't affect relevance and so I think it took us about 3 or 4 months to get that upgraded internally has as part of that effort we upgraded the open source genome and that was surprisingly easy and because at yeah because we have this stack and tree pattern we basically just had to change OK in the tree instead of doing this we do that and so I can point you to the poor quest for that but I was actually shocked how simple that was already well I thank you all very much for coming and I Jia on Twitter and I've got the slides of on my website I use higher you know if you're looking for a new opportunity or if your company looking for candidates get online and give it a shot and or come work for us that's pretty cool company so thanks everybody thank you for the higher if