Norikra: SQL Stream Processing in Ruby
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 65 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/37620 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Producer |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
Ruby Conference 201431 / 65
2
3
4
6
9
11
14
17
18
19
20
25
27
29
30
32
34
35
39
40
46
47
50
53
55
56
58
61
63
00:00
Process (computing)Streaming mediaBit rateSoftwareOpen sourceData streamOffice suiteMultiplication signDiagram
00:39
Cuboid
01:00
Right angleMusical ensembleMoment (mathematics)Cartesian coordinate systemTwitterRational numberLine (geometry)Message passingInternet service providerFacebookMereologyMetropolitan area networkForm (programming)Computer animation
01:46
Metric systemLoginGame theoryMultiplication signComputer animation
02:16
Data storage deviceProjective planeMultiplication signPhysical systemCoefficient of determinationService (economics)Letterpress printingData managementResultantAnalytic setComputing platformGraph (mathematics)Data analysisMessage passingLogin
03:43
Process (computing)SoftwareOpen sourcePhysical systemDataflowComputing platformInternet service providerData storage deviceComputer animation
04:14
Wave packetStreaming mediaInternet service providerDependent and independent variablesBit rateGraph (mathematics)Projective planeMathematicsResponse time (technology)Extension (kinesiology)Algebra2 (number)Multiplication signComputer animation
04:54
Graph (mathematics)Visualization (computer graphics)CalculationStreaming mediaPlug-in (computing)Extension (kinesiology)MathematicsRule of inferenceService (economics)Logic gateBootingGoodness of fitComputer animation
05:38
Metric systemService (economics)MathematicsLoginPower (physics)Configuration spaceComplex (psychology)Query languageMultiplication signEstimatorComputer animation
06:21
Event horizonArithmetic meanComputing platformMiddlewareHacker (term)Service (economics)Cartesian coordinate systemWritingLoginQuery languageElectronic data processingData analysisMultiplication signSoftwareProcess (computing)2 (number)HypermediaForm (programming)Direction (geometry)Streaming mediaProduct (business)Proper mapSoftware engineeringNumbering schemeNetwork topologyGradient
08:11
SoftwareNumbering schemeOffice suiteMiddlewareOpen sourceService (economics)Installation artStreaming mediaProcess (computing)Server (computing)StatisticsError messageDigital photographyInterface (computing)Computer animationDiagram
08:48
Client (computing)Web 2.0Message passingInterface (computing)Library (computing)SoftwareInstallation art2 (number)
09:15
Demo (music)2 (number)Event horizonService (economics)Client (computing)Object (grammar)Web 2.0Process (computing)
10:29
2 (number)Event horizonOpen setGraph (mathematics)CountingShooting method
11:20
Event horizonField (computer science)Range (statistics)outputResultantBoolean algebraComputer configurationExpressionQuery languageStreaming mediaVideo game consoleState of matterMultiplication signBit ratePerturbation theoryCellular automatonNumbering schemeNumber
14:25
Multiplication signStapeldateiFunction (mathematics)Perturbation theoryLogic gateMachine visionGroup actionOrder (biology)2 (number)Streaming mediaResultantQuery languageoutputComputer animation
15:51
Java appletHash functionFunctional (mathematics)Query languageEvent horizonField (computer science)Process (computing)
16:53
Attribute grammarObject (grammar)Query languageLatent heat
17:20
Message passingServer (computing)Dependent and independent variablesProduct (business)CASE <Informatik>Connected spaceWeb 2.0Cartesian coordinate systemData loggerIntegrated development environmentLine (geometry)WebsiteComputer animation
18:37
ResultantSystem administratorData loggerError messageFunction (mathematics)EmailVideo game consoleDenial-of-service attackBootingPRINCE2FreewareMultiplication sign
19:31
LoginError messageCASE <Informatik>Process (computing)Sound effectDefault (computer science)Pulse (signal processing)Form (programming)Workstation <Musikinstrument>Arithmetic meanWebsiteComputer programmingCountingMultiplication signComputer clusterService (economics)Traffic reportingProgrammer (hardware)Query languageElectronic data processingWritingFlow separationVideo game console2 (number)System administrator
21:25
CASE <Informatik>GoogolEvent horizonLetterpress printingProper mapResultantGoodness of fitNetwork topologyProcess (computing)Boss CorporationGrass (card game)Computer architectureStreaming mediaServer (computing)Metric systemWeb 2.0Web serviceRow (database)Service (economics)Query languageCloud computingLoginSpreadsheetDirectory servicePhysical systemGraph (mathematics)
23:48
Process (computing)TwitterLambda calculusComputer architectureSoftware developerLibrary (computing)Java appletComputing platformPoint (geometry)Query languagePosition operatorStapeldateiOpen setProgrammer (hardware)User-defined functionRepository (publishing)Set (mathematics)Complex (psychology)Event horizonStreaming mediaPlug-in (computing)DivisorDirectory serviceBasis <Mathematik>Electronic data processingExpert systemForcing (mathematics)Cellular automatonSoftwareComputer programmingGenderValue-added networkDifferent (Kate Ryan album)Device driverDialect
27:21
Category of beingCycle (graph theory)Query languageSoftwareComputing platformElectronic data processingWebsiteDiagram
Transcript: English(auto-generated)
00:19
Okay, and I will talk about,
00:22
it's time for my talk, so. I will talk about Norikra. Norikra is an open source software and I, that I love, and to process and data streams and the open source software written in Ruby. So this, and these are topics of today's talk.
00:45
And at first, I will talk about, at first, I will talk about and why and I love Norikra. And it is very important to understand what Norikra is and how Norikra works.
01:00
So, and my name is Satoshi Tagomori, also known as Tagomori, and that is my account name of Twitter and GitHub and many others. And I'm from Tokyo, Japan, and I'm working in line cooperation. Line cooperation is an internet service company is serving as a message application, line.
01:23
And line is a message application, just like WhatsApp or Facebook Messenger or any other. So, and line, line have an about 130 million users in worldwide and mainly in Asia and South America
01:41
and a part of Europe. So, and moreover, and we have many sub-services online platform, like in Japanese manga, electric publish, publish, publishment, and then camera and news, or Q&A service, or in weather news, and many games.
02:02
So we must handle a huge amount of logs and metrics and at the same time, and we must handle various kind of metrics and logs. So, and I am working about our data
02:20
and analytics platforms, and this is a very simple monitoring and data analytics platform overview. And at first, and we must collect many data from many servers, and pass in this data and clean up this data, and store in distributed storages,
02:44
like in Hadoop, HDFS. And then, and process the data and visualize it in graphs or charts on many others. So, and that is why I am a committer
03:01
of a Fruently project. Fruently project is, Kyoto Tamra talks about Fruently project, and two days ago, the first day of this RubyConf. And please check the slides about Fruently, but in roughly saying,
03:27
Fruently is a log management system and aggregate to collect many logs and to aggregate these logs, and to put these data and result data into any storages or remote systems.
03:44
So, and we are using Fruently in our data platform systems, and to deliver many data, and to control data flows. On the other hand, and we are using Hadoop and Hive to process our storage and stored data.
04:04
And Hadoop and Hive is a very famous open-source software, and many Internet service companies use these softwares. And then, and we must process the stream data
04:22
as soon as possible to show, to find any troubles or any surprising changes of our Internet service traffic. And like, HTTP response called in percentages, or HTTP request rate per second,
04:43
or an HTTP response times graph. And this graph is generated by a Fruently project. Fruently have a very,
05:00
Fruently have an extension and features, and the plugin features, and we can write and use any plugins to aggregate and stream data, and make percentages, or many values.
05:22
And we can put this data on graph tools, or any other visualization tools. And so Fruently is very good for simple data or simple calculations. But our services, and there are many more,
05:42
more and more different services, and there are many changes in a day, including in logging, and there are many kind of logs for each services, and there are many different metrics for each services. So, and Fruently, Fruently requires and configuration changes and restarts,
06:03
and to change what to do. And Fruently is not so good for processing about complex data or fragile environment, like data schema changes, or changes what we want to do.
06:20
So we want to add or remove queries any time we want. So what we want, we will change it very frequently. So we want to add or remove queries any time that we want and we want to write many queries for service log streams,
06:43
or we want to ignore events without data we want. And data schema will be changed very frequently. But the application engineers cannot know what requires,
07:01
what data process platform required. So they, so we should create a system and that the application engineers can change their log schemas and their logs and meanings any time.
07:23
So, and data analytics platform can or should be able to ignore events without data we want. And there, in my company, there are many service directors and growth hackers, they are growth hackers. So they are not software engineers,
07:42
but they know what is important for growth of our services. So we want to make our service directors and growth hackers to write their own queries for what they want.
08:01
That is why I wrote in Norikra, Norikra is a data processing platform and middleware to realize these requirements. Okay, Norikra, Norikra is a schema-less stream processing
08:23
as middleware and with SQL. That is a open source service software and within JRuby and runs on JVM. Norikra is distributed in RubyGems.org, so we can install Norikra, just do gem install Norikra,
08:43
and then we can launch the server by Norikra stat. And Norikra have some interfaces like in CRI client or client libraries named as nruby-client.gem. So we can operate Norikra with these CRI commands
09:04
or we can also control the software and over web UI and over HTTP API with JSONs and message back. So let me show some demo of Norikra.
09:22
Okay, and we can install Norikra by this command and already installed. Okay, and Norikra is written in JRuby, so this launch requires very long seconds. So, and here, this is a sample.
09:49
Okay, and this is a event example. Oh, and the JSON objects are within two fields,
10:03
name and quantity. So we can feed these data into Norikra over Norikra client and event send or my service sales. Okay, but this command and nothing happens.
10:26
So because Norikra requires, okay, this is in web UI, and Norikra requires and target definition. So at first, we should, oh, we should define
10:41
the targets, target opens. And now I'm cheating, so in this shell, I'm using in C Ruby. So C Ruby does not require several seconds to launch the commands. And okay, now I will feed log events
11:08
and continuously into Norikra to Norikra. Okay, so Norikra can find and field names
11:23
from these event streams, like a name and quantity. So now we want to select these fields from this event stream, like by this and very simple SQL.
11:43
My service, filter, and to specify to print and query results into console. Okay, successfully added.
12:00
And we can get, now we got the result data from result data with a name and quantity from this and event stream. So anytime we can change the input data schema. So now, this is another example
12:25
with another additional schema. And now, I will feed this data to Norikra, but the previous query works well
12:44
and the previous events. But schema is already changed. Okay, and drunk Boolean optional fields detected automatically. And now, and we can write any SQL to, for example,
13:06
we can count any input events by this, oh, where drunk.
13:22
Drunk is, and Boolean, so this expression is correct. But this SQL is not correct for Norikra. Norikra requires the range and how long is the aggregation rate, Norikra requires
13:43
an aggregation range. And what, and okay, with this specification, Norikra can count their events for every five events.
14:00
And Norikra can count their drunk, events with drunk is true, and in every five events. Okay, and query is added, oh, oh, oh, there is too many
14:21
and record, so suspend the previous query, okay. We can get an output data, oh, time batch. Oh, I mistakes at, okay. And also, we can count in any,
14:44
and some of quantities by this, and very simple, query, name and summarizing. Summarization of a quantity from my service, sales.
15:03
And group by name, order by summarization, descendant. And the get and summarization for every five seconds.
15:37
Okay, and we can get even some aggregated results
15:44
by an SQL query for this input stream. That is how Norikra works. And we can, so with Norikra handles and schemas event stream, and we can add
16:02
or remove data fields whenever we want. And Norikra uses SQL, and Norikra requires a normal restart to add or remove queries. And Norikra's SQL, with Norikra SQL,
16:22
and joins and subqueries are available, and we can add and use the defined functions and written in Ruby or Java or any other JVM languages, and we can publish that UDF as a Ruby gem.
16:40
And Norikra can handle nested hash and arrays, and these values are accessible directly from SQL like this. So user attribute have a JSON object,
17:01
nested JSON object, and attend is a nested JSON array, but Norikra's query is extended, so we can access user.age, or attend.data0 or any other specifications.
17:21
Okay, and we are now using Norikra in our production environment, and the first-hand use case is the error log summarizations and we have a web API for our partners.
17:49
Over that API, any user sends messages to our partner's official accounts,
18:02
and then our server sends these messages to our partner's server, and that is written in a Business Connect server at the right side, and then our partners respond their own messages to our API server,
18:22
and then we bring that response to our users in our application line. But our partner's server, if our partner's server goes down,
18:44
if our error messages, if all of our error messages and brings to our partners, and that is the really and flooding. So to avoid the flooding of error logs or error messages,
19:02
and we are now summarizing these error logs and error messages by this SQL in Norikra, and then Norikra puts the summarized and output and currently sends an email to our partners,
19:21
and at the same time, Norikra saves these results of summarized log into MySQL, and then our administration console shows that the summarized logs to our partners.
19:41
And error log summarization is a very major use case for Norikra. On the other hand, Norikra, we are using Hadoop, and at the same time, in our data processing platform,
20:02
and we can process the same data and with Hadoop and with Norikra. And our service programmers can write queries on Hadoop and on Norikra for just the same data.
20:25
So the prompt, these features are used to generate and prompt reports and daily fixed reports. And for our services, our programmers are writing
20:47
and these queries are hyped for and fixed reports and each day, and also writes queries to for Norikra
21:00
for our two producer prompt reports. Prompt reports are generated every one hour or several minutes or several seconds. These prompt reports are shown in administration console
21:20
for our customers of other services. And this is a use case by Google engineer, and Google, the cloud platform service architect of Google Cloud Platform.
21:41
And he uses Norikra with Google BigQuery and to count the web service requests and web service responses, and to show these results on dashboard on Google spreadsheet and Google Apps script.
22:08
He uses a web server, NGX, and NGX writes and it's access log into disk. So, friendlies are executed on each server
22:24
and each reads an access log and sends it to Norikra. And Norikra summarizes these access logs per server and the summarized records are sent to BigQuery directory
22:41
and another aggregation node. And this is a total overview of this system. And the summarized data are collected into an aggregation node, and then aggregation node can count the whole status of these events
23:03
within another Norikra aggregation node. And then friendlies writes these results into Google spreadsheet, and Google spreadsheet shows graphs and summaries by Google Apps script.
23:21
If users want metrics not already defined, and users can throw queries into BigQuery. And Norikra and BigQuery can process
23:41
just the same data set. And this architecture is called as a Lambda architecture and named by an engineer of Twitter. And Lambda architecture handles batch processing
24:01
and stream processing. So we can use SQL-like DSL, both in BigQuery and Norikra, and we can build an Lambda architecture platform with Norikra.
24:28
Why Norikra is written in JRuby? This is the main, there are big two factor, and one is ESPA. ESPA is a set and complex event processing library
24:41
written in Java. And ESPA provides an SQL-like DSL, and that is the base of Norikra query. And ESPA has been very well written library,
25:02
and we can process very huge amount of data with ESPA. And RubyGems.org, and RubyGems.org is of course an open repository for public UDF fragments, user-defined function plug-ins of Norikra
25:21
and provided as RubyGems. JRuby, and I don't, before Norikra, I do not use JRuby, but these two factors makes me to use JRuby.
25:45
So JRuby is, JRuby for me is just Ruby, and this is brought by great JRuby developer team. And JRuby makes developing Norikra dramatically faster with an ESPA and well-known Ruby's positive points.
26:07
And JRuby with RubyGems and RubyGems.org for an easy deployment and installation. And JRuby, and with JRuby, we can use Java libraries and like JRuby or ESPA
26:21
or many other libraries. I'm using many other Java libraries in Norikra. That is a very good point to build data processing middleware. But then my point is there are not so many JRuby users,
26:42
especially in Tokyo. And of course, we can find many JRuby committers in Tokyo or in Japan, but not so many JRuby programmers. We cannot find not so many JRuby programmers in Tokyo. When I got confused and call any Java method
27:06
or any other processing, but then I cannot ask these questions for not so many people. But then this is not so big a minus point that JRuby is very great software, I think.
27:24
So, and this is a wrap-up. And I believe that if you are interested in Norikra, and please check the software's URL documentation site or software URL, GitHub.
27:40
But then I believe that Norikra brings in our data platform. Norikra brings in more queries and more simplicity and less latency to our data processing platforms. So if you are interested in Norikra, please try it.
28:02
Okay, thank you.