Using Ruby in data science
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 69 | |
| Author | ||
| License | CC Attribution - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/37809 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
Ruby Conference 20178 / 69
2
5
8
11
13
19
20
21
32
33
34
35
37
40
43
44
52
54
55
56
57
58
62
64
65
67
00:00
Software developerTwitterStatisticsComputer programmingArithmetic meanPhysicalismXMLUMLComputer animation
02:06
Electric currentPerspective (visual)Ocean currentCycle (graph theory)Projective planePerspective (visual)Arithmetic meanPattern languageCartesian coordinate systemDiagramPhysical systemWordWebsitePresentation of a groupSlide ruleComputer animation
03:47
StatisticsSample (statistics)Matrix (mathematics)View (database)Projective planeSemiconductor memoryTensorSet (mathematics)Library (computing)Metric systemSoftware developerComputer animation
04:42
Library (computing)Numerical analysisMatrix (mathematics)Module (mathematics)StatisticsAlgebraMeta elementKeyboard shortcutFloating pointDisintegrationKernel (computing)MathematicsDistribution (mathematics)Prime idealCuboidInternet service providerTable (information)Frame problemUsabilityData structureTask (computing)DialectMetric systemProgramming languageForm (programming)Sparse matrixOcean currentOperator (mathematics)PiDifferent (Kate Ryan album)Process (computing)Linear algebraImplementationObject (grammar)Integrated development environmentNatural languageGeneric programmingStudent's t-testSpacetimeTwo-dimensional spaceAmenable group2 (number)Library (computing)Computer animation
07:06
Cluster samplingComputer fileMatrix (mathematics)BenchmarkCellular automatonView (database)Kernel (computing)Codierung <Programmierung>Array data structureSummierbarkeitElement (mathematics)Parameter (computer programming)LoginMathematicsFunction (mathematics)Electric currentVariable (mathematics)Menu (computing)Element (mathematics)SummierbarkeitNeuroinformatikPoint (geometry)InjektivitätMetric systemSystem callParameter (computer programming)2 (number)Different (Kate Ryan album)Symbol tableArray data structureComputer animation
08:08
BenchmarkMatrix (mathematics)Menu (computing)LoginPhysical systemLibrary (computing)Plot (narrative)Total S.A.PlotterMatrix (mathematics)Multiplication signMeasurementMetric systemCodeResultantInterior (topology)Library (computing)Run time (program lifecycle phase)Position operatorComputer animation
09:00
Kernel (computing)Computer fileView (database)BenchmarkMatrix (mathematics)LaptopCellular automatonFunctional (mathematics)Multiplication signRun time (program lifecycle phase)Metric systemNumberSummierbarkeitSoftware bugComputer fileComputer animation
09:48
Software developerFunction (mathematics)Matrix (mathematics)ImplementationSoftware developerTensorLibrary (computing)Formal languageProjective planeFunctional (mathematics)Task (computing)Data miningSparse matrixNumeral (linguistics)BefehlsprozessorFrequencyMetric system10 (number)Content (media)Coma BerenicesMatrix (mathematics)Arithmetic meanFrame problemComputer animation
12:35
Numerical analysisMatrix (mathematics)Arrow of timeSoftware developerCore dump1 (number)Slide ruleUniform resource locatorVirtual machineNumeral (linguistics)Core dumpArrow of timeFile formatMultiplication signProjective planeDeterminantPlanningSoftware developerMetric systemCASE <Informatik>ImplementationCoordinate systemReading (process)Product (business)Fiber (mathematics)Text editorFormal languageBitSemiconductor memoryResultantView (database)Error messagePattern languageCuboidComputational scienceComputer simulationNoise (electronics)System callComputer animation
15:30
Data analysisExploratory data analysisCodeMereologyKeyboard shortcutLibrary (computing)Object (grammar)Communications protocolGateway (telecommunications)Web applicationLibrary (computing)MereologyData miningObject-oriented programmingPower (physics)Process (computing)Phase transitionStandard deviationInterpreter (computing)2 (number)Functional (mathematics)CodeExploratory data analysisRuby on RailsKeyboard shortcutCASE <Informatik>Formal languageGateway (telecommunications)Core dumpMachine learningPoint (geometry)Musical ensembleTransport Layer SecurityInheritance (object-oriented programming)View (database)Bit rateLogic gateComputer fontComputer animation
17:51
Hydraulic jumpWrapper (data mining)Computer fontMereologyLibrary (computing)Wrapper (data mining)Module (mathematics)MultilaterationOnline helpStructural loadPower (physics)Computer animation
18:57
Cluster samplingComputer fileKernel (computing)LaptopView (database)Library (computing)BenchmarkElectronic visual displayComputer configurationMatrix (mathematics)Menu (computing)LoginDemo (music)CodeShape (magazine)Array data structureTelephone number mappingKeyboard shortcutSummierbarkeitStatisticsLocal GroupFunction (mathematics)SineMetric systemParticle systemModule (mathematics)Element (mathematics)CalculationRun time (program lifecycle phase)MeasurementSummierbarkeit2 (number)Multiplication signArray data structureExpressionResultantInjektivitätFrame problemBenchmarkMonoidLibrary (computing)System callGroup actionMultiplicationQuicksortNominal numberState of matterDivision (mathematics)Matrix (mathematics)Data miningServer (computing)Endliche ModelltheorieComputer animation
22:34
Matrix (mathematics)Menu (computing)LoginDemo (music)Functional (mathematics)Frame problemResultantMetric systemComputer animation
23:04
Telephone number mappingMenu (computing)LaptopDemo (music)PlotterResultant2 (number)Metric systemComputer animation
23:31
BlogOptical character recognitionForestRandom numberComputer-generated imageryStandard deviationData analysisSlide ruleForestProcess (computing)Ruby on RailsExploratory data analysisPattern languageCartesian coordinate systemINTEGRALStandard deviationCodeUniform resource locatorPower (physics)BlogFunctional (mathematics)Materialization (paranormal)Formal languageTask (computing)RandomizationShared memoryRight angleParticle systemGodArithmetic meanSampling (statistics)AnalogyDisk read-and-write headCausalityIPSecSystem callIdentity managementComputer animation
26:49
ImplementationSystem programmingDatabaseSigma-algebraArrow of timeCore dumpMereologyFile formatCore dumpImplementationPhysical systemPattern languageFunctional (mathematics)Row (database)Device driverArrow of timeDirectory serviceDatabaseEndliche ModelltheoriePerspective (visual)MathematicsElectronic data processingCartesian coordinate systemBuildingSerial portProjective planeINTEGRALLattice (order)CodeFrame problemRAIDMusical ensembleWebsiteProcess (computing)Operator (mathematics)Shared memoryCuboidComputer animationDiagram
29:35
CloningChainKeyboard shortcutPerspective (visual)Physical systemProcess (computing)Computer-generated imageryKeyboard shortcutRun time (program lifecycle phase)Projective planeRewritingVisualization (computer graphics)BenchmarkLaptopWeb 2.0Arithmetic progressionTensorPattern languagePhysical systemCloningMedical imagingMereologyCartesian coordinate systemElectronic data processingProcess (computing)Roundness (object)MetadataChainSeries (mathematics)Presentation of a groupSampling (statistics)WeightFreezingPulse (signal processing)Game theoryComputer animation
32:25
Row (database)Coma BerenicesXML
Transcript: English(auto-generated)
00:12
Hello, everyone, thank you for coming. I'd like to talk about using Ruby in data science.
00:21
So how many people related to data science? Hmm, not many, okay. So how many people do you want to use Ruby for data science? Oh, so many people want to use Ruby for data science.
00:45
Me too. Even for us working with Ruby, the opportunities to be involved in data science will increase more and more. But now Ruby hinders us from becoming familiar
01:02
with data science because Ruby is difficult to use in data science. However, it hasn't been so until recently. The situation is changing. Currently, Ruby is getting easier to use
01:21
with data science little by little. Did you know? No? No one knew it. But don't worry, no problem, I'll describe it in this talk.
01:41
So today I'll talk about how we can use Ruby in data science. Before that, let me introduce myself. My name is Kenta Murata. My handle name is M-R-K-N. Please call me Kenta or my nickname Muraken.
02:01
I'm working at the SPE, as a full-time CRB committer. So this is my company's logo. The company name SPE literally equals it to the successor of SPEED. It means that it's faster than faster what SPEED means.
02:28
In other words, the company iterates its business to try our cycles in overwhelming speed. As I mentioned in the previous slide, my company employs me as a full-time CRB committer.
02:44
I'm permitted to do any great things for Ruby ecosystem. In this year, I'm mostly working for making tools for data science that are used with application written in Ruby.
03:01
My talk consists of these topics. At first, I'll talk about the current situation of Ruby in data science. Next I'll show you the patterns to use Ruby in data science. Then I'll explain my perspective of the future of Ruby in data science.
03:20
Finally, I'll conclude my presentation. So let's start the main topics. The first topic is the current situation. Now, there are three major projects for data science in Ruby. This diagram shows the relationships between them.
03:42
So let me describe in more details about this project. The first is SciRuby. Do you know SciRuby? Not many. I think that SciRuby is most famous project
04:01
for people outside of Japan. SciRuby is a set of many gem libraries that use in metrics as they are in memory tensor data. It has many gem libraries, but many of them are dead
04:21
because their development has been stopped. You can check all the gems under SciRuby in this webpage. Let's check it.
04:44
So, as you can see, there are a lot of libraries listed in this page, but unfortunately, so this red and yellow is not usable for now. So half of them are marked dead.
05:08
I think there are three benefits when you use SciRuby gems. The first benefit is you only need Ruby. So, in other words, you don't need to prepare other programming language like Python.
05:24
The second benefit is you can use sparse metrics within metrics, but the current implementation doesn't support linear algebraic operations such as PCA for sparse metrics. So you cannot use any metrics for generic NLP,
05:43
natural language processing tasks. And the last benefit is you can use data frames with Darrow. Do you know data frame? Data frame is a basic data structure
06:01
to manipulate and visualize living data in data science. It is a two-dimensional data structure like a SQL table. In Ruby, Darrow provides data frame, so we can use data frame with Darrow.
06:20
As I'll describe later, you can also use data frames by using PyCall with Python environment, but while you use Darrow, you don't need to use Python together. SciRuby can be usable, but it also has several drawbacks.
06:44
I think three drawbacks, I think there are three important drawbacks in SciRubyGems. The first drawback is that in metrics it's extremely slow. So let me show you this problem in the demonstration.
07:25
Can you read the text? First, to show how any metrics is slow,
07:40
the following calls compare the three different kinds of summation methods with 1,000 element arrays. One method is any metrics is sum. Second is array sum, introduced in Ruby 2.4. And last is array inject with plus symbol argument.
08:00
At first, I should require the dependencies. And this code measures run time of each method.
08:26
As you can see, this end matrix's result run time is very large. So drawing a plot to visualize the result
08:45
by using rbplot library, so in metrics sum,
09:08
consume this amount of time. But others cannot see, because in metrics one time is very large.
09:20
As you can see this chart, in metrics sum is tremendously too much slow. This bug is filed as the issue number 362. So if you want to investigate and fix it, please check this issue and please send your request.
09:55
Back to the slide. The second drawback is about DALU. DALU can be usable for basic data manipulation,
10:04
but it lacks functions which pandas supports for practical data science tasks. So I strongly recommend you to use Python or R language if you need to do data mining in your business.
10:22
And the last drawback is that Sylluby is less documented. So it is hard to use when you are a beginner of Sylluby. The reason why there are three toolboxes due to the small population of developers and users,
10:41
so Sylluby always welcomes your contributions. The second project is Ruby Newmo. The founder of this project is Masahiro Tanaka. He is the original developer of the old original NRA.
11:02
It is the first in-memory tensor library for Ruby. Almost all Rubyists who need to manipulate tensor data in RubyScript use the original first NRA in ancient period. Since 2016, Masahiro Tanaka started Ruby Newmo project
11:25
to rewrite the old NRA for supporting the latest Ruby and the external libraries like OpenPlus, and to realize his new ideas of the implementation.
11:47
Like Sylluby, you only need Ruby, ah, I'm sorry. Ruby Newmo has some benefits and drawbacks. Like Sylluby, you only need Ruby for using Ruby Newmo.
12:00
And Newmo NRA is faster than any metrics, and pure Ruby, so in my opinion, it is the best library for manipulating in-memory numerical tensor data on CPUs. But Ruby Newmo does not support sparse metrics
12:20
and data frames. It means it is hard to use Ruby Newmo for NLP and data science tasks. And Ruby Newmo is also as documented as Sylluby is. If you're interested in Ruby Newmo and want to know the details of Ruby Newmo,
12:43
you can access the English slide and the Japanese talk movie in Ruby Kage 2017 at this URL. Please check them. I think you may want to know which Sylluby or Ruby Newmo is better.
13:04
The answer is case by case, I think. If you want to use Ruby for data science without any other languages, Sylluby is better because you need to use data frames, and Sylluby has Darwin.
13:21
If you want to do just scientific computing, such as numerical simulations, and try to implement your own machine learning algorithms, Ruby Newmo is better than Sylluby because NRA is faster than in metrics.
13:46
And the third project is Ruby Data Tools. This is based on Apache Arrow and its Ruby binding, Red Arrow. Red Data Tools is a very young project.
14:00
It started since this February. But as of today, it has these five gems. The biggest benefit of Ruby Data Tools is that you can try to use Apache Arrow in Ruby. Additionally, the core developer of Ruby Data Tools,
14:25
Kohei Suto, is a member of Apache Arrow's Project Management Committee. And this means you can continue to use Apache Arrow in Ruby in the future too. But there are two drawbacks.
14:40
The first is that gem of Ruby Data Tools is too young to use in production. So you should have a strong determination to employ this for your business products. The second drawback is now Apache Arrow is just a data format for in-memory and streaming IO.
15:03
So you cannot use it for manipulating data. Now you can do only load, save, and converting data, but they are too much faster than other way. Apache Arrow has a plan to implement
15:21
data manipulation APIs, so this drawback will be resolved by time. As you can see so far, it is hard to use data science by only Ruby. And almost all data scientists shouldn't want
15:42
to use Ruby in their jobs because they need the biggest powers of standard data tools in Python and R, such as Pandas and Spark, especially in exploratory data analysis. Exploratory data analysis phase is most important
16:01
for data mining and machine learning. The existing tools in Ruby aren't very durable for such use. But as you know, Ruby and Ruby on Rails are best for writing business web applications. So you should use Ruby and other language
16:24
like Python together. How to do it? I made a PyCall for such use cases. So what is PyCall? Using PyCall, you can use Python libraries
16:42
from your Ruby code very naturally. PyCall consists of two parts. The one part is the Ruby binding library of libpython.swell. It is a core of Python interpreter. Another part is a gateway between Ruby and Python
17:00
to translate their object systems. By the first part, PyCall provides us to access to the functions of Python interpreter. And by the second part, PyCall realizes the natural feeling for us Rubyists in the use of Python functions.
17:25
Let's look at the simple example use of PyCall.
17:53
I'm sorry. Too small font. I skipped this slide.
18:01
Don't worry, I prepared a demonstration for later part. Currently, I made wrappers for Namookpipe and Mataprot lib. But also, I want to make wrappers for scikit-learn, seaborn, bokeh, keras, and so on.
18:24
I need help to increase supported Python libraries. If you are interested to write such wrappers, please write your own wrapper, publish on GitHub and tell me.
18:42
By the way, PyCall also provides features to use Python libraries without writing wrapper libraries. So you can just load Python libraries as modules and use the modules. Let me show you a demonstration of PyCall.
19:23
In this demonstration, we correct the result of benchmark in pandas data frame and visualize it by seaborn library. At first, we needed to prepare requirements.
19:40
In this demonstration, I use nmatrix, new monoid array, and pandas, and mataprot lib.
20:01
And this is the benchmark call. In this benchmark call, I use 10,000 element arrays and calculate summation 100 times for each method.
20:26
The first method is array inject. The second is while expression. And the third is array sum. And second is enumerable sum.
20:43
And fourth is nmatrix sum, and the last is new monoid sum. Okay? Let's go.
21:01
In this call, measures runtime of the method by benchmark.realtime. Directly store into the pandas data frame.
21:22
So here I get the results in pandas data frame. This is the result. Using group by method calculates statistics,
21:43
this summaries for each method. As you can see, nmatrix is most slow,
22:02
and the second is inject. And then I visualize the data frame by seaborn's function. Using pycall.importmodule method,
22:23
we can import Python libraries as a module. So this is the result. So nmatrix's result is very large,
22:40
so we cannot see other method. I want to show only other except nmatrix result using these data frame as functions. Okay?
23:07
This is a bar plot result, nmatrix's result. So most of first method is array sum, and the second is an array. This demonstration is finished.
23:35
So this demonstration is very simple, but there are other resources I used previously
23:43
to show pycall examples. In RubyKaigi 2017, I demonstrated Keras examples and Rails integration example. Also, in RubyKaigi 2017, I did Ruby data workshop.
24:01
These resources and materials are available on GitHub. I've already uploaded this slide on my speaker deck, speakerdeck.com slash mrkn. So you can access these URLs from the slide.
24:21
Please check it. And there are other example users of pycalls. So there are two blog posts about scikit-to-run examples by Soren D.
24:41
The first article is simply users of scikit-to-run, and the second is OCR with scikit-to-run's random forest crossfire. And in kv-ruby conference in this year,
25:07
May held the workshop about pycall. You can see the workshop materials by this link. Please check them.
25:23
So as you can see in the demo, pycall provides us access to the functions of pycall data tools. So you can use all the following tools from your Ruby code. They are all the standard tools in data science.
25:51
So far we've learned about benefits and drawbacks of SciRuby, Ruby New More, and red data tools. And we've learned what we can do with pycall.
26:02
From here, I want to show you the current best patterns to use Ruby in data science. As I mentioned before, you should use Ruby and other languages like pycall together because almost all data scientists shouldn't want to use Ruby in their jobs.
26:22
And they need the biggest powers of standard data tools like Pandas to do exploratory data analysis. It's most important task is to find valuable knowledge from living data from business. And also, we won't use Ruby and Ruby on Rails
26:42
for writing our business application because it is best for us. So I propose three implementation patterns to integrate application within Ruby and data processing system within Python.
27:05
The first pattern is referring the same database directory from both systems. This is very easy to implement, but the changes in application may affect data processing side, especially about the data schema changes.
27:23
The second pattern is calling the functions of data processing side from application side. To implement this pattern, we need to serialize data to pass it from application to data processing system. So, large serialization cost can be occurred.
27:46
The last pattern is using pycall to call the function of data processing system. We can write the driver code in Ruby, so we can share the active record models between application and the data processing system.
28:02
Using the pycall, we can build Pandas data frames directly from Ruby. So, no serialization cost in this pattern. We need to choose the right way according to the situation.
28:23
The last part of this talk is the future perspective of Ruby in data science. I want to explain about two topics about the future. One is Apache Arrow, and the other is GPGPU and Deep Learning.
28:41
Apache Arrow is efficient data format and going to have efficient data manipulate operations. So, I think it will be the core of almost data tools in the future. It's already decided to replace the core part of Pandas 2.0 with Apache Arrow,
29:01
and PySpark already uses Apache Arrow for exchanging data between Python and Spark. So, the red data tools project is important for the future of Ruby's data science system. If you're interested in Apache Arrow, I strongly recommend to join red data tools project.
29:27
You can access the red data tools project at the URL. Next is about the GPGPU.
29:43
About the GPGPU, as Presenland presented yesterday, we already have array fire for use GPU. Moreover, two GPGPU projects were accepted by Ruby Grant this year.
30:00
One is RBCUDA by Presen. That is binding of CUDA runtime rivalries. Another Ruby Grant project is proposed by Sonac, one of the Ruby committers maintaining logo. He will make CUMO. That is a CUPI clone for NUMA NRA.
30:25
And for Deep Learning, we already have TensorFlow RB written by Arafato. Moreover, there are two work in progress projects I recognize. Red china is a part of red data tools project.
30:42
This is started by Hatapi to rewrite china in Ruby. This project uses the NUMA NRA for tensor data, so it will be able to use GPU by CUMO. And now I'm working for writing Ruby binding of MXNet.
31:03
I think it can be released after a few months. Finally, I'll conclude my presentation. In my presentation, I describe three major projects in Ruby about data science.
31:21
Say as high Ruby, we need more on the web data tools. Next, I demonstrate PyCall with an example of benchmark visualization. Moreover, I restulate the three patterns to integrate application between Ruby and data processing system written in Python. Finally, I talked about a future past
31:43
picked about Apache role, GPGPU, and Deep Learning. And we prepare the Docker image that contains almost all the things to try Ruby's data science ecosystem. So you can run Jupyter notebook as my demonstration
32:08
in a Docker container by this command. Please try it. That's all. Thank you.