Logo TIB AV-Portal Logo TIB AV-Portal

Using logging hooks for real-time log analysis

Video in TIB AV-Portal: Using logging hooks for real-time log analysis

Formal Metadata

Using logging hooks for real-time log analysis
Alternative Title
Use logging hooks for real-time log analysis
Title of Series
Number of Parts
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date
Production Place
Ottawa, Canada

Content Metadata

Subject Area
If you want to know right now what your server has been doing When problems arise, every DBA wants to know what the database has been up to. If your best idea so far has been to parse the log files or load the CSV logs into another database, you might want to learn about a better way. With logging hooks, you can attach data sinks directly to the PostgreSQL server log stream and analyze logging events as they happen. Many useful applications arise this way: put log data into another PostgreSQL database for ad hoc querying store log data in Hadoop for analytics throw log data into a graphing system generate alerts directly from logging events I'll show you how to put these things together and use them in practice.
web pages Slides bottom code log time singularities materials First-Order similar sets help analysis web mathematics internet Hardware Query Video file system box level errors platforms Multiple generate real moment sign in basis volume Transactional Demo connections words means processes Computer animation case disk website metrics spaces
point files Development attitudes log time real singularities First-Order analysis events Kraft photos round Query errors form formating real Development counting attitudes CAN-bus words Computer animation Query statements log files Video spaces
fake Computer animation URN
point Super files code time maximal sign in sun Arm variance van Emulation mechanisms completion Rolling structure message-based modules man Super management load inner Databases functions scans CAN-bus hook message-based Computer animation case URN inference ensemble sort Hacker
unit server implementation code load component sources analysis volume Databases production CAN-bus processes Computer animation string Right
modules files link patched van versions CAN-bus Computer animation repository normal model extent Sum
Meta time sun mathematics completion different Rolling law information Drums Arc man satellite inner instance processes Void's URN Now orders configuration 12th control files patched neural network 3rd maximal Average Emulation van SICS Sum CAMS unite plugin Dean-Zahl Gravitationsgesetz unit CAP management sin directories CAN-bus algebraic Computer animation statements localization libraries Vacuum
unit CAP files namespacing network OS moment time zone maximal 3rd argument sign in sun ones CAN-bus Pointer Computer animation URN configuration WPAN unite WAN Gravitationsgesetz extent
dots NET unit time zone 3rd argument smart sun area Em's message-based Arc form man CAP unit hard Menu Airy Functions parity variables statistics CAN-bus Computer animation case URN extent clone
network OS neural network ones 3rd maximal Average sun Arm fields programs van variance Kommensurabilität completion string testing input PEAS Sum unite Development Kit Scalable Coherent Interface Gravitationsgesetz area man unit sin formating inner Amsterdam Ordnance Datum ups gute directories degree sign CAN-bus processes Computer animation URN Right
web pages icons man unit implementation PINs Venn diagrams Tiled log 3rd Amsterdam Ordnance Datum Ext ups infinity part sun total variance templates CAN-bus Pointer Computer animation URN Sum
code open subsets argument sun variance different errors systems man width formating moment construction inner Menu maximal part data management processes Multi-Agent URN configuration input Right Sum WAN web pages control mapping network OS maximal 3rd division van Emulation number production level integers encoding CAMS Sum addresses Gravitationsgesetz Super addition unit demon information demo Magneto-optical drive Databases inclusion CAN-bus words Computer animation Blog Universal services table localization
suite concurrent code directions sources cloud platforms loss data analysis sun Arm information Sehne <Geometrie> 19th mathematics different localization partition Partial Scalable Coherent Interface man satellite Development inner Ext instance meetings Regular Expressions statistics several connections URN Sum WAN mud machine maximal Average van production loops terms string telnet ensemble level Sum optimal Gravitationsgesetz Installer comparison unit Multiple sin cellular analysis plan Databases gute bin system call ones CAN-bus sign Computer animation case HMS <Fertigung> statements iterations table libraries Vacuum
Home Page server services Java projects storage sets scans van backend web category Computer animation case orders website
Serialization code sources combination argument sun variance permutations Pointer MKS-System different box toe Arc classes man programming language formating moment storage effects Menu nominations scans processes URN Now figure fitness source spaces web pages Slides PINs files network OS 3rd maximal fan van second versions voting naturally structure rules addition unit torus sin plan gute bin CAN-bus inclusion Computer animation
flow PINs files visualization maximal 3rd cloud platforms sun van Emulation ease of use dual Pointer completion configuration unite classes unit sin inner Amsterdam Ordnance Datum print CAN-bus Computer animation URN configuration simulation
point code Ionic gaps mid 3rd maximal functions infinity sun Arm number sonic Pointer Subspaces rates fuzzing localization errors unite systems man unit Transactional several entire CAN-bus Computer animation PIE URN HMS <Fertigung> WPAN metrics
ASP Actions services study neural network print part sun number rates structure platforms man man relation Graph moment analysis Stream bits CAN-bus Indexable Types words Computer animation software system mathematicians
web pages point building link fatigue time patched component sources shape part events fan theories workloads hypermedia string data fusion level statements plugin partition unit link relation forces Development load digital bits Databases statistics characteristics Indexable hook words processes errors Computer animation software repository statements sort figure table localization
Computer animation real log Query First-Order analysis spaces
will translate things this page is in Tagalog so nothing so if you can understand them sorry about that
and I don't have a lot of slides and then use some the most but I'm told that everything here is being recorded in video and audio so you can review that later and I also have all the materials that are in the public was stories which I'll get into it so I am computerizing charged time process and you're I currently work at meaning which is a social networking platform I live and work in Pennsylvania and you can reach me on the Internet at least places and of course if you start from and so when I for reasons that will become obvious in a moment when I saw of started so working on this problem space which all get to moment I I thought of this as beyond but that last year there was already talk and not this conference but another conference was already toggle Yonkers along and there's so in in some sense this is so that the next generation after that perhaps so I I won't mouth my thing here says I'm assume your data architect but in practice for the most most of my time and the and don't we so far as I mentioned in a social network or social networking platform depends on how you want to put it so we have you know regular interest when Web infrastructure backed by by post set of basis which are there said questionnaire what so what volume I'm I'm not and all that but I can that someone help here so I can so we know we have a website open to the internet we do you know deployments multiple times a day so the whole thing is in in constant motion and really deviously you have to keep the data the the bottom of that you're running and you know we have montessori set up in every so often something similar alarm triggers and with the fraunhofer triggers right now but does doesn't so um and in my mind on in the mind of our team there are at least 2 kinds of people a lot so interesting 1 this so the what I think of maybe hardware level words when the the boxes died on that file system is gone read only on disk has disappeared and things like that and not I can use of and not and in those cases the passes off to after the of scene and they know you have to repair the box or or repair the disk things like that so we don't really we're not really concerned about that we have the normal standby infrastructure so you fail reference if needed and if things go on the more interesting and more problematic alerts are when certain metrics change and and the metrics that we're interested in for example is things like connection found transactions per 2nd errors per 2nd so you load 0 wait times those little things and and and those can change what kinds of reason that the website is getting hammered don't new new things were deployed that you know maybe don't work as they should in and things like that and then we need to go in and and and find out what happened and that's what this is about so what 1 thing we actually have 1 show use GIS code was avoided did not well we have this
will be called the developer attitude readjustment to called the short got just this
thing here that's our that's our
so if you if you're a developer and you could select that that's what you no
that's that's that's the that's take as an example which is easy to understand the city you are your monitoring tells you that the you ever errors per 2nd counterpoint up and and now if you wanna know why so the know straightforward way to do this is you go into lot files and you grab around and after while crappy even you know it would be be nice if you could put this into a more structured form and then we can also use a real query language on this so in many many years ago do you see is the lot format was created for that purpose ostensibly and then you yeah that that's the reason why idea but for these various reasons that doesn't really work for us 1 is that just the time to take the log files and millions of wealth analyzes this that takes too long and we have to be at this infrastructure of set up for that but it just it didn't know it was possible so you know if you wanna know what they are coming up have to fix when it within a few minutes I I can't wait for you know 5 minutes 10 minutes for all this stuff to the topical words and loaded an indexical the the other problem with that with that is that the if you have ongoing events you want you want attractiveness they go on so if you take a walk on commas or take a look and see what happens then what would you do with the data that came in and there's a couple minutes and you don't need to take the safe way you needed the the more left often and take the next round of the log file and and do all kinds of stuff and and that's a very complicated I know some people do but it's still not nice just as a side note if you saw is getting hammered don't rotate the log files that takes so which we've tried it once and to try again so there are no other solutions in the space that you probably heard of something like you better Piaget in features that statements those all the tools that are related to that you know think they don't really solve this problem of like what what was the count what was that what was agreed that through the most errors in the last 5 minutes what was agreed that how did actually behave yesterday at the same time and things like that so that that did really help us so limited looking around became across a a overlooked teacher 9 point it was present which is the
logging hoax and this like so voting also available in posters might want to so you can use and today all all of this stuff I'm describing here is what we will be using in the real world that like all hopes and post this they're not document and so that's where the gets overlooked but if you
were have here the I have all this in the material will but just to show you the this
commit that can be so this this was committed 9 point you by the end the 2 years of work the future came from this fellow clock who mentioned on here who works for Skype in his music he's the guy who actually wrote the yes at stuff that I get blamed for 4 years ago so I have some of the most electronegative feature supposed so when the war what it does is is pretty simple if you look at the
data and so huge commentary which is actually the documentation and actually just call introduces a new variable here MIT log book that emit log book I support variable which you can be assigned to function and that function it's already time log messages written to the the text file that you would otherwise have and so if you would if you In a load a module using the normal mechanisms you can execute any code you want at that point and so now we have endless possibilities right so going back to the use case what what I wanna do is take D is the log data and throw it into a structured sort for example database so that
basically what I wanna do something
something like that right so I have my production in the base year I use this hope and then some extra code which are all get to and then you can to another database and you can do the analysis there and all this is in not cause I real-time might not actually assigned but as it happens as a string so you could know right you in book for that it's not really hard but I think everyone who no seasonal talk about packing process so the idea here is to use existing components to you to do their work and Mozambique but also wrote a lot of different you can consider the reference implementation for lot which is this a tool called InterLoc caught you can search for a 100 euro all the nodes but you just circulate your final and get help the as far as I understand Skype's motivation like this whole future is not necessarily what I'm describing here but they were concerned about that just plain logging the log volume takes their source talent for it has too much load on the server so they wanted to set wages have the the logging off quickly but do not lend itself to other users as well so and now all of that show you how that works so here's the
here's the it have repository for so you don't have to read the link don't of and actually in In the patches subdirectory there so there's the actual longing book patch is in there so you can actually back at all the way to a point for and then I I've lost on that so that's that's reversible as a solid not big so you know if you if you want to use this in all the versions of it's new 1 your consisted install patch and you can use in future other than that it's just 1 simple see file and you you build a like a normal pitch excess module which gets me to the so I have this all of this stuff and you as I have that on on my data so you you can play with that later also if you want come is a so the 1st diversity 1 you is build a and have this always a you from the analyzes the wonderful extension model which is really annoying so you need to do this thing around so install so was is now this is a one to here so that's always look different so there is 1
can its agreement because it's executable not that's another flat from the from does not now the the way you the way you configure the US is as follows so this is a process
configuration voluntary stuff in and then at the end you add since statement so there are 2 ways to load 1 1 is using shared libraries and 1 of using local pretty libraries and difference for those who don't know it's shared preload libraries have loaded into the postmaster time to start the instance and local through libraries loaded at time you start a session and I just you through changes that sentence example that I recommend you putting it into local preload libraries for the reason that a you don't have to restore to struggle to even install this and to you know keeps all the kid keeps the stuff out of the cosmos which you want to have stable but you but you don't have to do it that way you can also it to show the in order to In order to stuff from short-read libraries have to do the following and you need to make a direct across plug-ins here and then move that thing in and that's only bizarre and I'm trying to get rid of that and those who follow all the patches I Centered resided that's the reason why I introduced control which didn't get into 9 3 to have so they a white list of things you can put into the culture local preload libraries so in the future September the solution and so now that that's insulation isolation you need to do this put into the blood and directory if you wanna loaded through local freedom libraries although it is the way it was and now we take this configuration file that I just showed as it as an example and to put in
place reload start size so what what this nodes in the background document traits and log log so as will see something in a moment skip back to that file so once once
you have loaded 1 way or the other you have this lock forward namespace of new configuration parameters and and have no
but of that has and and you
can create multiple destinations but this is just a simple case of you 1 destination user party names variable variable here which you just invent a random enhanced and then based on that you you have additional parameters that get calibrated on the fly once it reads this year and then you give it the destination of where this supposed to go and what did lock or does it it sends these log data as UDP packets to the destination is that you get here in the form of that give unit supports J. so long and that string and so so so if you just like keep using so as long can use that against Jason is the most interesting because it during the civil structured form and and you know that they made the this is the destination we have the agent receives that and then you need a put so once once you have the running
you can you know you can use netcat to contestants you well it and did you often when only supports J. sciences and that string but in it's not hard to add more formats and that's you should also you know consider this as a tool kits to some degree I think the very simple as it's fine but if you want something different you just need to this is the simplest as as a string in so that out of unity so so
here's the here's the UDP packet being sent around so that's all the typical logging fields that did you sure you're all familiar with in the adjacent format and now you can imagine what you would do that right you will thing and process on the quantitative Jason and then you have the warning format that all the log data and can do whatever you want so India so you probably need to do some programming of so in the PGA lock forward repository there's also area in the testing directories actually solve an example
implementation in the part which
kind of serious or if you want to no it's not long to pages you're actually handles Jason that's answers look so this is the long when you really need to be and so this you can also use a tessellation like character and
from of the stuff and then you can actually use that as a template and interview so in to that and then we just show you how you would
do multiple destinations so here's a different conformational changes to stop I just showed but as 2nd destination here so this is the stuff we already saw and here I just into the new target name which uses in a slightly different destinations local managers different words and there's also an additional or in a small number of additional arguments we can prefilter the stuff that the data is processed by the so this is not very user-friendly some things perhaps addresses that you can give the minimum error level as an integer which is the internal encoding of the what is it the blog info warning so that that's kind of the universe is not going to be doing anything from easily change so this would be this would be you can then write a 2nd agent which just gets the errors in the format and then you can look at it maybe in the learning system or it a system that Council of something things like that so a guess it's partially obvious the reason UDP is used here is that you don't want they this whole additional processing the whole of the actual session it I have experimented with other logging hopes that the example using the MQP to put it this information to Cuba then if your broker Dallas problems in the every of the q everything so UDP in that sense this is useful but of course this doesn't this is not going to be in auditing system because it can be used controller right so now we get going back to the original goal was putting them into a database is then not difficult for me so that a new you can put it into the same cluster if you want then you have kind of open problem you don't configure this correctly and and practicing from 1 of his in a different process so make a
separate cluster here end product database very small demo schema again all this data is online you can try this so which yourself so now I just created various we also the table moments and I wrote it essentially took the download python code I just showed and put some database stuff in there so instead of just finding out it takes the sum of the log and input everything in here right now but in some along the edges the table that has previously beyond is literally 1 page approach almost let's start that is running this of course was also don't have to sort of and occasionally construction so
with any kind of we have a warning didn't table and this is essentially reversible what I'm actually used in practice now if you look at that there's no way you can imagine all kinds of things that you can improve so 1st of all you want change there the level thing to something you can read but for example here durations are being a lot here you and for example 1 thing I do the actual inside you know run this through regular expressions split up 2 calls so I directions of the columns from the statement and then I can run In analysis on the 4 iterations and that's or if you are interested in something very specific you specifically interested in this connections or connections you can take the sums without the manual and so on so that's it essentially you know the solution of partial problem know of of course you also want petition the stable maybe normalizes to split out the actual strings majority in the repeated interviews at separate tables of manage source so way and the way the way I run this I start to use this agent on and supporters of the and that way you can also stand multiple instance quite easily because this thing is not in the area concurrent or is not currently spite of support variously so you can spend on a bunch of different instances given the important on boston . different places at different production sources from watson's that you've concurrency that there it's not very pretty but it seems to work just fine so that's 1 doing 1 thing I'm actually a playing with that and that and terms of concurrency is really implemented that agent I just showed him and go to programming language which has better conferences support and I what but don't get out so some of the experiment with go can look at that and they all this is not what I actually using production but it's something on the plane but to make this optimizes process the now this this is again this is something connection everyone you can just install new score or up something that suits their legal use case something that came up during the development meeting yesterday's to perhaps you know right a prepackaged solutions here that if you more easily when you don't have to write any code yourself so that might be the future direction and describe the different locking hopes that long straight back into the same instance you know avoiding loops and stuff like that of course and so for small instances in which a perhaps just you know loaded and use it without having to do any extra work there might be something to it to try and while you that advantage of doing loading it and cost from libraries is that you can actually set up for a database we use also so that might be useful but of course 1 of the advantages of not having the same instance all 1906 poses it can also aggravate the the data from multiple machines so you have to have multiple genes belonging to the same table for the partition cells tables and compare things that way so that is also definitely advantage of this some of the other solutions it just never look at a single so but now that's sad and yeah the best senseless also may use case of you know what happened the last 5 minutes if you want a single the long-term perhaps you you want basically keep all the loss and you have big data analysis later also here have for example through the stuff individual and have some things to show how to do that so
getting back to this also picture
made so in the set of I'm
using I'm still using digital orders the books are not writing on properties of the agent and using something called Apache flume so that's an Apache project which is that
from a budget Oregon has a nice picture of a home page which looks you know somewhat similar to what we want to do and so on flew describes itself a distributed reliable available service for efficiently collecting aggregating moving large amounts of log data so that seems to be purposeful for this whole thing being an Apache project their main sold primary use cases having web server log data put into the big data store but it doesn't have to be worked on this side and you know it has also to be achieved from this site but it's kind of geared toward putting things in there as a front-end to each of us essentially but the the same on the on this can be pretty much anything and its supporting entailing a lot while throwing in here or getting since located data all kinds of things or can write your which is what I did so always is Java so you you don't just standard the trough and to contact you can place to go the
if you were the soul of dig deeper into this space of moving structured data from 1 end to some of the structural storage this about you know 8 different formats will encounter 5 different programming languages and you have to somehow pick a combination that is fully static and this is the least amount of data on the way and I'm sure there's other permutations of how getting the data so 1 of the most straightforward way to do this would be take data and look my which fumes supports out of the box and the wrote this is all data into food or you autistic if you want and then through this into a us on structural and then new year's clicker or highest to to get it out and that that would work was not really what I set out to do so I wrote a few pieces of code of my own which so it would seem that takes that Jason P. format on accident into influence native internal transport format and then on the other hand take on the other side takes that structure format and throws it into H faces as could separate columns essentially so the way you configure figure flew has as its sources and sinks so here's my a there's a bunch of course you can use that just a few innocent example here's a source that I wrote which not not published at the moment but I don't plan to do that is to say set to some internal but this is not the rocket science so I have running this code on this forward which should look familiar from the previous example that took a job for for publishers is also additional features for years and has thing called interceptor which FIL Mendel's the data and 1 intercepted can use for examples to throw times and into the existing data because doesn't privatized is we and and on the sink slide here is an example for saying using the Bolton nature that support which just throws data in there unstructured way into HDFS files and if you don't know what to do with the there's actually talking in the next slide you want now it actually it's it's still rotates followers that once it just throw stuff into files and then afterward this effect additional parameters you can tweak death of what files creates and what they're named and then you would use something like which is sold transparently wraps all that you can make a lot of you want so that's 1 example and the other what i'd concentrating on using age-based this is the H based thing is stripped of blue but it doesn't actually know how to unpack the stuff on sending and so what is different from serializer so there's a bunch of job classes in in despite and have to the you know how do you want to get the correct so just to show that as well the fisheries works today just Table 1 call the exciting so in well this is actually running locally on the left half so it should be that's lovely RH nominations started storage for this is also the sort if if you don't start page basic installing start to challenge the kids go equivalent but you can't do anything so you it's it's very interesting lots of version 2 . 4 seconds of intelligence also so
so this flow has not many options are actually quite usable start agent Due to the class of stuff to get your customers sure in the have to cure config file which is the 1 I just showed you and then you have to total which agent you and let's see that
works I
so this is dividing output from my code so you can see it's already doing something and entire
Lokpal something over here OK this pointed out we can do stuff with pictures so if you if you 1 play ways billion data point in that this is a way general lots so so that's basically where uh as I mentioned in the 1st half of the talk of putting lot they'll and posters is 1 using this is something I'm looking into actively now the data itself something that I originally wanted to do but have not found a good solution for yet is to actually you know some of these the actual rate monitoring using this and in the beginning I mentioned I'm looking at metrics like number of errors per 2nd or number of transactions per 2nd and and that itself has been computed by a separate kind of legacy monitoring system and ideally this you know if you have this kind of data you can throw it into some of the system that can do the counting for you but found good want to do that yet so is a couple things that look there was some thing called rock-steady traveling for
which also comes with a nice picture but has lots of moving parts so you can read all the small print so this is your 1st this is relevant to this is what study this graphite is not this is 4th so there's no way to men who reports you don't have to you have to monitor the monitoring essentially so that is a bit to do for me at the moment but maybe you know a lot of effort on make this work another thing that I'm actively looking at is think alters the money like the
mathematician from the man that I know that the would actually do exactly what I needed that you can have managed to set up yet but new going something to look at the next couple months and of course graphite you can also use animals also looking into throwing a lot of data into graphite just looking at a graph of the action the the rates monitoring of graph I I have haven't found that to reliable but gives you some useful numbers but if you want to directly to the monetary it's not exactly so that user that at the moment so that said there are a a growing number of commercial new platform word software and service type solutions out there is no overlap a lot with all of these problems such as new relative or strength 1 of the other so you want throw it supported and despondent soldiers to the full text indexing and you can do the analysis based on that but that doesn't have any knowledge of the structure of the post was lost that is not you know exactly what you want and the new revenue relics also something using but it it doesn't have any knowledge was was uh it's it's still getting there but it's not the but perhaps only a few years when we look back we don't have to do this anymore but we don't know of course all the solutions also a huge amount of money social wanna spend the money and this is a toolkits for you so To conclude so look
into logging works and look into the dual-port you think you you can probably use that and if you if you're if you have been wanting for additional features in the pages that statements maybe you need to wait learning can you build Avolio you and and I suspect something so here are coupling so if in theory in theory I have all this material to get a repository which in theory should be linked from the you do con website it wasn't as a couple of hours ago but I think it's 2nd depend upon once in a while so eventually it will show up the we can you you find it through me if you if you don't find so the link to all the tools you know you know here's the modern book committee which essentially sorts of documentation if you wanna look that up the future looks forward tool I showed you the thing I started writing and draw if you want to experiment with that link to the room and nothing actually about looking into its throwing the log data into a rabbit and you so I mentioned I try to write a lot of books that publishers straight into MQP sort than that because of the blocking issue but there is a the UDP plugin for and Q which is a small amount of tweaking you could actually just throw the UDP data fusion local publishers into that and even if you women it and don't have to write your own agents and stuff you just write consumers and use I I am told by our residents relative Q X for that there are some architectural problems with that you few exchanges is implemented I have given the time which is scarce 2nd fixed so but that's certainly very interesting and then into what that is what I have any questions just all word it does it does have a very rudimentary ways of thought so the question for the audience Georgia asked if the beautiful awful doesn't have any way of filtering it does have very simple ways of filtering what I showed by the dialog level geologists it has a way to filter by it's just a text string it looks for so you can do small amount of filtering that yeah if you wanna go beyond that of course you can you change the source approach the question and that then becomes how much lower you wanna put into that component verses of the downstream components of thing as a universal and that as a mention Skype so of building this as a way of offloading load from a source of the problem that can be very interesting that that that says you mentioned wanting does take each patches to digital forces and few fixes and many features so that certainly in active development you know there are certain probably wasted can add little bit more filtering but if it's too complex problems with that someone downstream this the license appeared for problem as many things on GitHub doesn't have a license
actually has a license that that's license of all you can see the letters but we can get from the shape of was same now in it that it it's fine it's fine to use it the the yeah OK so if you if saying you wanna use triggers on the year logging database to like of events what that was the 1st part 1 and they use this to write it it down the and what the yeah OK so that would be that so you're saying you wanna put triggers on the lobbying table to look for forward that based on text a lot of things to look for events and put them in other places yes OK yes certainly having the possibilities here and most importantly the reason why I want to show serious to just have more people involved in this these or ideas to see what kind of a common ground with the and 1 cavity there is of course produce a lot of what data you can also overload along database size so I would you know the that has entered with lots of extra figures stuff on on the receiving end and perhaps have more asynchronous proper processing that you have to you have to find out for yourself but it's a possibility young because 1 of the more people in all these things so as you the yeah but it's not different from like how those that's collected works at some point at some point you have to wait all you have to elucidated in only after that's the trade off you have to make for itself I I have you know what I'm using expenditure amount to new workload and network and all that stuff so it seems to work well you I would like to do in throw more data and never have enough on sweet spot works and we all so we have media partitioning and no indexes cost the basics of you know or know with the business costs I get my my business partner perhaps you
yeah but you feel free to come up afterward if you want to stay in contact and you United the people who know me know where to find me if you don't know where to find simplest send e-mail it ought to be or I would certainly like to chat more people about what they their interest in this space to to don't more common solutions OK there's a few inches and you doesn't obsession next next lots of of things