Bestand wählen

Using PostgreSQL with Redis

Zitierlink des Filmsegments
Embed Code

Automatisierte Medienanalyse

Erkannte Entitäten
OK good morning everybody has 1st at that you get to work usually if I the results of the my allergies applying someone so we're gonna talk about pets crests and readers and I will get will look at
it here is basically what we're gonna cover we're gonna talk about a bit about what readers is and how it works then we'll take a look at the the red as foreign data direct and then we'll look at the companion package which wraps readers command API and finally we can have a look at a case study of of a half of form Z so that's based on the use of combination of espresso and written and uses his utility readers is a
high performance in-memory key-value data stored that does it is well actually write this can result from and how does that is is configurable and also has some replication features which I won't be covering here for time but essentially this is this is what it is it's it's a pretty simple have data stores the nice thing about the the thing about it that makes it attractive for use with the with measures is good is that it does have some structured data such as we'll see the
it has almost no configurations require the set up that so commands you need to do on a Fedora instance recently uh reason fedora instance to installed run at on and get into talking to a glass is just command line client talking to read that's and and that's all you have to do is pretty much now we get to production them you're going to choose a bit principally how much memory it uses but that to to start off with that's all you need to do very very simple reduces
as a sensitivity data store In the case of just strings all and for the values of a rather different they
can be Skylar values that can be strings all they can be integers so you have increment operations which is useful for doing things like web cameras and so on and they can also be structured and these are the 4 structures that they support the list of things unordered sets of things have ordered sets of things and and in fact these actually stored the the data you can insert a value into these things along with a sum score which can then the ordering is according to the score and you get back at school and ends at but then the last thing I have is as a single value can consist of a set of name-value pairs uh somewhat like a make yeah the pretty much I don't think we're actually doing any of that kind of think of this um so and these these hashes play a big part in the way that we're using the rest to reducing conduction with with Chris this
is a very simple command said that it's schematic is unfortunately quite lied but all commands recent was nothing like uh like SQL and its basic you a command and a few parameters that's the URL where you can go and have a look here are
some examples so the 1st example is sitting in dialog values and which they said the name of the the name of the object and the value of object the the 2nd command is sitting some values on a hash key so this is simply an arbitrary 90 minutes at all a property name value how the 1st is adding some value to for is adding some values to the end of the list and the it is setting some values In order to this the the the score is by the way can be here floating-point numbers that have to be integers which OK there is no
creation command for a readers of it simply at springs into existence when you when you add some value to it so it's almost scandalous the only thing is that if you try to use a list operation on a set or something like that then uh then dates but apart from that chair readers pretty much tightly scheme right and we have a
single survivor man for all the objects all types they just have to objects of different type of same every object has to have its habits have a unique name so you people tend to build you meet names by no along with sort of pattern well I that 1 of the things that that 1 of things you can do is this page is this command keys which will find all the all keys that match a particular pattern that using type bobbing type patterns the radius that was uh the red as 1 document actually warn against using this demand in production and you will see later on hello to get around uh this this problem by using a different a command find the case that we can make so British
users basically due in effect tables that say segregated book the global namespace into particular as portions by on the other they use a prefix says that although they adopted a date on although precision idea and part of the key or something other which is going then segregate that and then they use use the keys command that we talk about or what they do is is this they keep the genes in a separate set so you might say have a 9 sickle key hits it you'll get educated that and then if you wanna gun final keys that are in your state that using the case command you'll simply get that the members from this set which is a very fast operation and process that but the downside is that the application has to make use of that itself is unable to match up the members all of a set or indeed the values returned by by the case commands to anything else you see your application has to do all work the but
the there is a client library which the radius community has produced called high risk pretty simple and it's available on GitHub um the greatest foreign data around it is something that page right to start with several years ago and so on back in the the 9 Monday's removed the starting off with that with that better wrappers and you recently it's been brought up to date the other runs with those with nonzero and somewhat extended by me and I have recently we moved it from personal it had space into this more generic names as and that's what next that is now the authoritative source for the foreign direct the the tip of the by the way is apparently broken what about out of i'll tell you about the the breakage and uh when I get there we want to leave the latest stuff of them in a while OK so
originally basically it only support valid that have simply ignored everything else and so it's not a scam on I'm not going to give you back into the and it had no support for segmenting the the namespace in so basically treated the whole radius namespace as a single type and that's the only thing you could do you could say well I want I want just that 1 key that would actually push down the hall from from post this on on the the name for recall a quality I only and if you did that then would actually go and that key and not give you back or as otherwise it would give you back all the rest of up as this apply anywhere conditions but of course that not mean you get a whole lot of data back from wrist and then you and you having to fill among the postprocess so study ran last December I
started to doing some some of these updates the 1st thing I did was to make it work with Postbus 9 2 and then I we started with all the touch and supported so you can have a table that will deal with set objects or list object or hash objects ordered set of objects as well as with static objects the 2nd thing we did was to provide an option we could specify the prefix for a set of keys for the table so then your table is and you know if you say that the prefixes now uh it's then I will only fixed keys without with names that matches that of the prefix all or it can use the table set functionality that we talked about before you provide the name of a city and then the keys of the table the values contained in that set but if the diet can if there is for all look fairly except skyline the data comes back as an and that section 2 and as opposed post a right literal so if you actually define the table it is considered as being a can and a writer takes that then it will actually break that break that up into uh right which you can then index and we use that in a variety of ways against
has tables of the most important at because really there that most closely to and participants records but the post this record is essentially the a mapping from a set of named column names the set of values so Aridus hash is exactly like a road map so if you take if you using a half part table then the best thing is to define the table as having a riot in the 2nd column and then what you can actually do is it's possible to turn that into a day song or I explored the to a record so here's an example
I wouldn't take this will create a font terrible cold which sections which is going to have a key always kind pretty much have to have 1 or 2 the columns a king and the status and or by a text was in the table type is called hash and we using the prefix methods to to choose which case we want and we can simply select staff from web sessions and will get a bunch of stations 1 right position so to use that with I store what we can do is we can make because I still has the ability to 10 an array of key value pairs into Oracle and we could create a record type for instance which has these properties ID browse renews and and then we use a expose populate record to turn that into records all of this stuff makes pretty well and there is
also a new the the package extension which I recently right called Jason object which pretty much that's the same thing as H those summary uh populate record but it doesn't from so this actually it turns since this into the adjacent object that if you if what you want is adjacent object rather than I'd still a record you can actually create 1 from and of course there in 9 . 3 we have a function that and J. into records however that's part of the new Testament somewhere
but so here's the thing don't ever use the key prefix in production the ad server that we're going to talk about they they they they actually tried that was the 1st pattern I want to use and they did make less of civil without what a ball would have indicated he's and he's an economy a couple weeks later said but it's not performing very well as it will have what what what's going what happens if you try but we that said and suddenly magically just would really fast all to do was it was as there were no setting the values they would also do as they had on the the and then when fetch that it was just very fast this is the pattern that was actually recommended by the red box they say don't use these use a set of keys OK
the other thing you can do this because you can you can actually use a set of keys is a kind of way of filter on on on a table so you can exceed the 4 you can put the please you want in the inredis said the finer table would use that set as he thing and then pull it back in that effect will subset you all the data that you you want and they're actually using this in there so it works pretty well we don't actually have support for high food for a lot of push down still in the in the farm data represent unique you get need to use these sorts of tricks at at least at the moment and if somebody wants to wants to try and push down we can have a look at it but and this we don't have any plans on the table for the OK so as I
mentioned in 9 . 3 we've got Jason populate record which means you could avoid these weights for the 1 thing that it is a record post 9 this problem probably be a good idea to have something which didn't actually have to use these intermediate product a on alright will create a recording we what we should have a function either in Khorasan extension which would have taken a right key-value pairs instantly produce a record of the required at and that would be quite possible is probably with the sectors is probably with having intercourse of the of the Fisher linear general use would certainly be useful for use with the rest the but
is the thing I write recently which uh and which is currently cause the better rapidly be broken some the from the rest foreign direct apparently treats every based on every readers and the values set I table right but what if they actually treated as being a title instead of a around and the classic case for doing this would be set so so what you can actually do is when I well would you will be heavily when I and break this it is defined a a this table to to correspond to a single radius objects and then get the values in that object back as a set of right that has all sorts of possible use useful that said the video section lists will soon become back as a single field raise passes the gonna come back as a key value rose to use with the fields and ordered set so you can you you can choose 1 or 2 fields I future if you have 2 feels the 2nd is going to be a sport that corresponds to the value in the ordered set
and ticularly after my experiences the last couple days are now known had a right the writeable foreign better rebels after the black hole experience and the the not that that require a lot of cash and there my on planning to start with this soon on having the restaurant they're able able to right bell used back to read the but in the
meanwhile we can't do that we simply can't do that because 9 2 i which doesn't have support for writeable foreign better repres anyway still what we've of what we've done is we've wrapped that that library that which is also used in foreign data the high rate of library we wrap its functionality in some post functions which a fairly simple this work was sponsored by the CEC usually to the creators of the the the observer and it's and available on their big back at reading and it
basically has full functions rest in a reduced discarnate risk manner is me the rest foreign direct makes a command for every time you select it doesn't it doesn't keep persistent connections hanging around but this is really designed for much lower level operations that we didn't want be had been to connect and disconnect every time you can push a single value you might want to was a million values from the table and you might have of that 1 of the having to make a connection on and break a connection for every every that every single value that I was watching the red so basically what you can do that what you have to do we're using this library is to set up a handle is in red is connected then use that handle in the restaurant and red from and I agree plans and then when you finish you just yeah final the the interesting arbitrary not of unless you really useful the no I don't know of anything in the city of foreign direct right the mean basically it's up environmental once you once you got that you you you from the point of view of a post Christian just basically got a table so everything has to be handled inside the foreign direct I don't have anything that you would give you XIs to arbitrary amounts so even when we get rotable writeable foreign direct as this probably still going to be some value in this library because you're going to be able to yeah do do things like flash forwards you want to to delete to clean up the database and that sort of stuff and some to to to get access to the utility commands a red OK so the 1st
argument to rest and is a handle its basic it's about 300 to which you choose it's a number but it needs to be a number between 0 and 15 and simply an index into an array of it's extremely lazy way to do that and so apologize for that not probably ought to fix it making the fence and 9 and then look that up in a hash table but I didn't write to those in a hurry so the all remaining are optional and all fold to something that very so normally what you want if you're running on the local host and you're running on with with red z that readers that bases also 9 0 3 things that I and if there is you can simply given the handle name you want and then use the defaults ignore duplicaat says if we've already created a handle with this but with this data that object to arm and that was found useful by obviously in and some of the curve occur as I as
I mentioned that the of persistent connections so you need to set them up and break them down explicitly so that these
other things on our from basically a very thin line over the the library functions with which have almost identical names on the difference if you want use 1 just use rate command that the rest command is the first one I write a but unfortunately mapping and this doesn't work terribly well with a very the functions and so risk command where you actually passing with of vector of arguments the metabolic an of what without tonight about knowledge and that which a whole lot better and so reduce command you can only have 4 arguments up to 4 arguments after monster if you need more than that then it objects at the moment that takes values but I'm not switch switches from using variadic takes variadic I was clear all that familiar with variadic any at the time I wrote this originally and this is becoming we were done 1 or 2 things with a particularly in the day some space and it has certain limitations but then think that particularly applies in this case Tom and that would make it a lot easier so you would have to have things to text I just found that with 1 of the other utilities I wrote that it was a whole lot better when we use them again OK so
the users outputs data inredis and altered run utilities the and it's been used for both of those things in their heads so I'm
set of also entity in an the ability to and will push a record and basically what this can with this will do it will pull a data out of the recall without actually having to 9 it explicitly and manipulate that you can tell us how you just passing the record you can tell if you want to push the keys which you normally do you took the case that you want the the king is added to you can tell what the prefix is going to be and what the what the what the the key fields are in in the record and then your record could be you know hundreds of variables long and you would have to do anything would just sit that this is basically for pushing a hash table is not and it was set up the the set for you as well and if you don't pass of knowledge here as well as all the hash where's that works quite nicely at the moment this is not in use that and they're actually doing some would be interested in inside an imperial biggest you well this is going to make life a whole lot simpler in the units of again so why use really well it's fast it's safely really fast but it's not terribly safer on this notion of transactions is almost nonexistent and where the extent its existence of nectar we represent so on Dudley stop in in relational terms said it's useful it's useful for certain types of data it's true if it's data that you can afford to lose or at least the lose a little bit of and then certainly useful was really and what the real reason is that given the constraints of that we were working on we could not have achieved the performance gap uh dolls by simply by stating all data impacts we would overwhelm
so the use case is an answer end basically attitude was you know if races if registration and sonicated even if file whole systems flight that is attractive because all our customers again to leave if we done so there is there really fast said this
is this project was that the granted by the sea it's with done pretty much on a shoestring and there are so small boutique and software has been here in North Carolina love with with with quite a few years has sponsored a number of extremely useful and this developments such as extensible in on the end of the bench with people so most of these slides uh based on information from the so here the system
dolls that they had on the need to be out of the 10 thousand at 2nd per CPU so they're actually lying on acorn machines in the database of its 80 thousand on the the using so all hardware require very cheaply that would they do not want to spin up beyond hardware and overcome the bloom limitations by running on superfast had with the timing constraints that they need to be able to answer this the system was designed to answer the the question of which had in 2 stages the 1st stage is extracted from Post and what it does is it says 1 of the edge we can all legally supplied to comply with this request and comes up with a list all around about 30 the 2nd of question it answers these also theory which 1 is is going to be the best the server based on the tistical analysis of what so recently been served in this particular area and which ones are actually going to pay the best because it's based on pricing in and out of there is a fairly complex algorithm which is applied that answer is computed from the data that is held in red it records the and requests the conformations and the clicked and it's a pretty much what needs to be a 24 7 operations
so here is a physical view of the system essentially we have these 4 systems at the bottom there for dull 29 50 which are not high machines right they didn't have a huge amount of memory and 2 of them is that run the system run run and they're all run put all these run Postgres databases tools and run the database and a hot standby for answering the 1st question and 2 of them run the the back system that I will have an effect that were the does the analytic ability and back office support Everything else runs on virtual hosts run from these zen has sort out on the right hand side and and that includes rates reduces running out there and radius is actually cheering up almost all of the 128 gigabytes on 2 instances on on each 1 on each of those is in house and runs with about 1997 gigabytes of memory again
is a logical view in this it uh of works on the the system is at the front in this but the adjectives said that genetics which deals with the fastest the poles of the dead straight of fossils the and the queries coming through Internet AS switches which were found with extremely fast it talks to post press at the top and then to readers here and answers the query and it's it's answering a query in about 10 to the end of 12 ms the query the the transaction database hold only 2 titles and I that's we actually construct that the tables every 10 minutes and the constructed in the business database with conventional LTP type system we construct that would push push data out to the transactional database highly be normalized form I will build the index is then we simply swap for the L run using arena and that provides pretty much no interruption whatsoever we have not noticed any slowdown at all as we're doing right which very well well so it replicates some data into the data warehouse from business database with the analytics and that is done because on selective basis using longest 3 of them so there we
have we have 6 this databases as having a pretty much just about cover this slide what I just said yeah all the other piece of this is that that we have PG pool balances over the transactional database and fellow in inifinity else falls under a it's not a tragedy but if if the transactional database falls and then we're not serving ads anymore we're losing money said it's and that tried Kuala stuff to to make it fall over the natural pulled plants and the and crash machines in various ways and it's still stand up that's it's fairly robust I
guess of business database as I said is is pretty conventional 0 OTP talk to the system itself the Philly normalized system will go tables for and some advertisers and publishers IP locations and and stuff is only a handful of users attached this there which is which I have business customers this set the people who publish ads and the people who advertise and but they actually tried in the 1st instance serving and 7 years and requests direct evidence that advice they did not get to 1st base they could not sustain anything like like that was necessary certainly not with the hardware they had on hand so we push we push these materialized views out up to in effect they're not to be answering them 1
of the keys to this is that we are using some very uh the we use has this advanced data types a lot I don't know the size of an edge is a post geometric out of the box and this is very what there is extensive use of arrays of various types we use some text vectors and take such queries out to match the words and we used to grabbing we've got just about every database and the pace support every in the extracted as support on 1 2 and and it's it all working actually extremely well 13 they don't know whether or not they can actually they tried stressing this and what on the current set up what happened was that direction oversaturate then it will part before there were able to stress is that it and of there was just what the
but this is the the at 7 databases I said we we simply copy the data in this very penalized for every 3 minutes that get a and create indexes and swap enough and they are absolutely convinced that this could not be done where has best extremely that the repressed this friendly and they 1 annotator thank you very much OK so we're getting
listening and 10 thousand impressions per 2nd all these impressions are actually getting into infrared and then that pulled from radius in batches using that using the foreign direct in the set model the uh every 6 minutes for statistical analysis on effect been doing this with the individual incident and stress would have been an overwhelming the user partitioning and the to the run monopoly back processing to produce the produce billing reports the statistical analysis ends again there are extremely grateful for the way the petitioning which with vectors that he works very well for
again at the end all 1 of the very impressions I'm not stored in direct in the past press but then that's not where the money the money in in serving ads is when people actually click on a map the data is actually recorded straight into the recording database as well as indirect but that's only a tiny fraction of of of the traffic because yeah you bring up a web page again 100 ads in most of most often you're not going to click on any said clicks than a Catherine at impressions out all almost all traffic but of course you might pay 1 at the thousand impressions where you might pay 50 cents or a dollar for a quick OK so basically the this recording database does all the old back-office processing on the initially tried calling this data from the data in the in the business databases required in using using DB-Link they found that was way too slow and said they replicate that in using titles and it's a it's a lot of and fraud detection for
the Texans actually and major issues in and there is a considerable incentive on publishers and generate clicks only only adds to their publishing and so they get some revenue and you need to be unable to analyze the the year the patterns of impressions and clicks In order to prevent fraud this is actually done in 2 2 stages firstly uh they have some flaw detection that is running it pretty much in real time against the right of database and uh where it where the clicks are initially stored but they also do a longer term of fraud detection in the recording database because basically a day's worth of of clicks at any 1 time of impressions at any 1 time in the red they claim that pretty regularly that means you need to store a lot of data in the recording the case we've
without pretty much this was telling us what we already know here that in in the ad server is a set of 94 128 gigabytes on each of those that has the decay rate and almost all of them you know this the anything that runs on a but the so that
found that this and that the stuff that we did was a lot faster than what you'd make life a lot easier falls on their and our benefit had the right artists themselves and they found good performance characters against
the stuff going the other way we actually need to 1 of the things you need when you make a decision about which at the center is what are the prices that are being paid for impressions and clicks because that helps you to decide which will use the view that all that and that that I that I forget pushed from the business database into relative that you can make the decision in red you know the all the rest clad in effect and make the it and they on the other hand the data in the impression input is pulled from readers in the recording database was that we have interaction both but it has
been in production since March going pretty well as i said that and it was it was done on a shoestring pretty much to you guys with some help from me did this in the space of 3 months said here the
conclusions rich datatypes absolutely essential could've done without which the annexes loop and the different index Help the interaction with the good interaction between reticent by this essentially we're using Red the do what raised as well and has pressed the what as a as well this that this is the key to good hybrid systems I'm convinced we can see more and more hybrids begin 1 told to do all where is probably I'm not going to cut it for a whole lot of applications in you get to see people combining different sorts of data stores for different sorts of use somebody else was telling me about using as this with Hadoop the other day and was seeing a number of cases of that itself no of it is very fast it's concurrency is is supposed to and they are that pretty much in love with and very then the costing they think that their development cost was approximately 10 % of what the about commercial ultimately would of course it's all source so that's it don't
you yet no j is essentially answering it all dynamic questions so the only thing that interacts with the that's serving the at the bed in other words the answer comes back from Nigeria this is this is that the 7 next which is about all the acts of queries handed off by In the next 2 days and the whole the whole F is written and I dataset then yeah the model right now common mom not sure what the what protocol is to tell you the truth but it's probably something fairly fairly much like that what all right case the references all the is the if you and to for you yeah not what used the when the so in that way so price the half all now with have source right question the race track and I have to In this way you know get an error at every all always commands check that you can talk to rest my from oppose this that so now thing comes in and says I have I got a connection the connections in the airport OK now on the run this command well if if if it didn't work on the log with the result back in Europe for right the 1 of yeah yeah I don't know added probably worth looking at to see whether we could on the trouble with the trouble with not not doing ephemeral connections is that we would have to have some API to to make and persist on that may be and you know there's nothing in the foreign direct the model which I actually which actually did that but I don't that might not be an objection we can you know it might be that we can simply sets of is that if there is a connection with use of if not to make something like that I think but the formation of the the seen or of which yeah we're not going to do it and maybe we could say that no use it could be an option on the on the foreign table in use persistent or whatever and then if if you hadn't been so that you would have happened if it had that you just the right you're going to have to actually and what do you think for yeah all yeah that's probably worth there no and it says that you know not now there is no listing of of their time at all yeah it is basically a key-value store that yes he was of you yet all right to the the other thing is that that kind not sure to tell you the truth I haven't written OK
Quelle <Physik>
Gewichtete Summe
Statistische Analyse
Ordnung <Mathematik>
Array <Informatik>
Automatische Indexierung
Boolesche Algebra
Natürliche Sprache
Generator <Informatik>
Ordnung <Mathematik>
Lesen <Datenverarbeitung>
Tabelle <Informatik>
Objekt <Kategorie>
Interrupt <Informatik>
Wrapper <Programmierung>
Virtuelle Maschine
Weg <Topologie>
Tabelle <Informatik>
Wort <Informatik>
Boolesche Algebra
Prozess <Physik>
Befehl <Informatik>
Lineares Funktional
Konfiguration <Informatik>
Konforme Abbildung
Automatische Indexierung
Projektive Ebene
Varietät <Mathematik>
Physikalische Theorie
Physikalisches System
Inverser Limes
Operations Research
Speicher <Informatik>
Einfach zusammenhängender Raum
Physikalisches System
Objekt <Kategorie>
Desintegration <Mathematik>
Charakteristisches Polynom
Trennschärfe <Statistik>
Metropolitan area network
Kategorie <Mathematik>
Güte der Anpassung
Rechter Winkel
Relationale Datenbank
Automatische Handlungsplanung
Analytische Menge
Überlagerung <Mathematik>
Open Source
Protokoll <Datenverarbeitungssystem>
Statistische Analyse
Elektronische Publikation
Konfiguration <Informatik>
Kartesische Koordinaten
Einheit <Mathematik>
Stützpunkt <Mathematik>
Nichtlinearer Operator
Funktion <Mathematik>
Ganze Zahl
Registrierung <Bildverarbeitung>
Gewicht <Mathematik>
Interaktives Fernsehen
Ganze Zahl
Wrapper <Programmierung>
Einfache Genauigkeit
Automatische Differentiation
Mapping <Computergraphik>


Formale Metadaten

Titel Using PostgreSQL with Redis
Alternativer Titel PostgreSQL + Redis
Serientitel PGCon 2013
Anzahl der Teile 25
Autor Dunstan, Andrew
Mitwirkende Heroku (Sponsor)
Lizenz CC-Namensnennung - keine kommerzielle Nutzung - Weitergabe unter gleichen Bedingungen 3.0 Unported:
Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben
DOI 10.5446/19065
Herausgeber PGCon - PostgreSQL Conference for Users and Developers, Andrea Ross
Erscheinungsjahr 2013
Sprache Englisch
Produktionsort Ottawa, Canada

Inhaltliche Metadaten

Fachgebiet Informatik
Abstract Using native wrappers and a Foreign Data Wrapper for two-way Redis integration Redis is a high performance in-memory distributed data store, which can work well in conjunction with PostgreSQL in certain types of applications. The talk will introduce native wrappers for the Redis API, which allow pushing data into Redis at high speed, as well as use of both the native API and the Redis Foreign Data Wrapper to pull data from Redis. We will also discuss possible scenarios where use of each of these is a win, and the trade-offs involved.

Ähnliche Filme