Jupyter for everything else
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Part Number | 149 | |
| Number of Parts | 169 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/21193 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2016149 / 169
1
5
6
7
10
11
12
13
18
20
24
26
29
30
31
32
33
36
39
41
44
48
51
52
53
59
60
62
68
69
71
79
82
83
84
85
90
91
98
99
101
102
106
110
113
114
115
118
122
123
124
125
132
133
135
136
137
140
143
144
145
147
148
149
151
153
154
155
156
158
162
163
166
167
169
00:00
Enterprise architectureProcess (computing)Online helpVideo game consoleLaptopKernel (computing)File formatBitUniverse (mathematics)Hacker (term)Different (Kate Ryan album)HypermediaHeegaard splittingHypercubeEinbettung <Mathematik>Web browserGroup actionMixed realityLoop (music)LaptopYouTubeVideo game consoleKernel (computing)WebsiteInterpreter (computing)AstrophysicsFlow separationFunction (mathematics)MereologyFormal languageEllipseSlide ruleCellular automatonWeb pageDebuggerLink (knot theory)CodePoint (geometry)VolumenvisualisierungSeries (mathematics)Projective planeLetterpress printingSinc functionCartesian coordinate systemRevision controloutputCASE <Informatik>Process (computing)Line (geometry)Level (video gaming)Beta functionPhysical systemProduct (business)Element (mathematics)Exploratory data analysisIntegrated development environmentRight angleDecision theoryResultantControl flowSoftware testingVideoconferencingComputer-generated imageryDemo (music)Enterprise architectureMessage passingInequality (mathematics)Computer animation
06:46
Maxima and minimaVirtual machineLaptopEndliche ModelltheorieModule (mathematics)SummierbarkeitMachine learningLink (knot theory)Data miningComputer networkProgrammable read-only memoryBayesian networkCheat <Computerspiel>ScherbeanspruchungInformationDistribution (mathematics)Server (computing)MultiplicationComputer-generated imageryDisintegrationPoint cloudBeta functionSoftware repositoryIRIS-TKernel (computing)Extension (kinesiology)GUI widgetWeb pageFormal languageIntegrated development environmentJava appletFatou-MengeAlgebraic closureChi-squared distributionNumbering schemeInstallation artSlide ruleLaptopRepository (publishing)SoftwareWebsiteFormal languageMeta elementElement (mathematics)Electronic mailing listIntegrated development environmentNumbering schemeInstance (computer science)Connectivity (graph theory)Point cloudKernel (computing)Web browserCloud computingVirtual machineCellular automatonGUI widgetExtension (kinesiology)Server (computing)Presentation of a groupDemo (music)PlanningSet (mathematics)Projective planeSpacetime1 (number)Different (Kate Ryan album)GoogolContinuum hypothesisComputer configurationCuboidWindowQuicksortGastropod shellThermal conductivityGradientSlide ruleDrop (liquid)Medical imagingTask (computing)StapeldateiCategory of beingComputer animationLecture/Conference
12:20
GUI widgetData managementKernel (computing)Gateway (telecommunications)Extension (kinesiology)LaptopServer (computing)Communications protocolClient (computing)Special unitary groupInterface (computing)Alpha (investment)CodeLaptopCellular automatonDifferent (Kate Ryan album)Integrated development environmentRadical (chemistry)Formal languageIntegral elementMoment (mathematics)GUI widgetInterface (computing)Alpha (investment)Data analysisKernel (computing)Gateway (telecommunications)Software developerMereologyLinearizationProjective planeComputer fileWeb 2.0Set (mathematics)INTEGRALDeclarative programmingCuboidSimilarity (geometry)1 (number)Real numberStandard deviationContent (media)Natural numberFunctional (mathematics)Complete metric spaceClient (computing)WindowModule (mathematics)Point (geometry)Table (information)BlogUser interfaceDemo (music)Electronic mailing listFraunhofer-Institut für Materialfluss und LogistikData managementRight angleForm (programming)Arithmetic meanDemosceneMedical imagingStability theoryFiber bundleRule of inferenceQuicksortElement (mathematics)Hidden Markov modelElectric generatorFront and back endsEndliche ModelltheorieComputer animation
17:24
Cellular automatonCodeBlogWorld Wide Web ConsortiumHash functionLine (geometry)Kernel (computing)Interactive televisionExtension (kinesiology)LaptopInjektivitätCodeRadical (chemistry)BlogQuicksortVariety (linguistics)Cellular automatonBlock (periodic table)Web pageForm (programming)Instance (computer science)Right angleWebsiteIntegrated development environmentHypermediaLecture/ConferenceComputer animation
18:25
Social classExtension (kinesiology)Cellular automatonRight angleProcess (computing)LaptopBlogGastropod shellStapeldateiVideo gameKernel (computing)Integrated development environmentConsistencyInstance (computer science)BitServer (computing)TelecommunicationLecture/Conference
20:08
World Wide Web ConsortiumKernel (computing)Hash functionLine (geometry)Interactive televisionExtension (kinesiology)BlogDefault (computer science)Web 2.0TelecommunicationComputing platformLaptopData managementPresentation of a groupStructural loadWeb browserPoint cloudPhysical systemBitEmailElectronic mailing listMathematicsOpen setComputer animation
21:43
Kernel (computing)Meta elementFamilyLine (geometry)MaizeLaptopSocial classTask (computing)Software developerDemo (music)WindowKeyboard shortcutKernel (computing)Metric systemMathematicsStapeldateiMeta elementLaptopSet (mathematics)Moment (mathematics)Presentation of a groupVideo gameNumbering schemeFamilyOcean currentMultiplication signElement (mathematics)Line (geometry)BuildingComputer animation
23:41
Wide area networkOpen setNormal (geometry)LaptopMetropolitan area networkSimulationArmGamma functionStack (abstract data type)Table (information)File formatFunction (mathematics)Special unitary groupSimultaneous localization and mappingExecution unitUniform resource nameMaizeHookingElectronic mailing listMaxima and minimaProgrammable read-only memorySource codeEmailWorld Wide Web ConsortiumSoftware development kitData typeSineMUDCuboid3 (number)Shift operatorConvex hullInformationCellular automatonComputer networkExt functorSummierbarkeitDilution (equation)View (database)Graph (mathematics)Information managementInclusion mapComputer fileOnline helpExtension (kinesiology)Plot (narrative)Sign (mathematics)Scripting languageJava appletMKS system of unitsDirected graphDreizehnSoftware testingMenu (computing)Interior (topology)InfinityJunction (traffic)Letterpress printingChi-squared distributionVisualization (computer graphics)Disk read-and-write headMathematicsAddressing modeDiscrete element methodLie groupKernel (computing)EstimationElectronic visual displayComputer-generated imageryNetwork operating systemLine (geometry)View (database)Programming paradigmElectronic visual displayGraph (mathematics)CASE <Informatik>Kernel (computing)Content (media)Computer fileFunctional (mathematics)ResultantLine (geometry)HierarchyMeta elementQuicksortGraphics libraryLaptopServer (computing)Different (Kate Ryan album)Multiplication signFunction (mathematics)Table (information)SoftwareClient (computing)Connected spaceStapeldateiScripting languageRadical (chemistry)WahrscheinlichkeitsfunktionExistenceIndependence (probability theory)LogicComputer animation
28:29
Electronic visual displaySpecial unitary groupSoftware testingExecution unitComputer-generated imagerySummierbarkeitNewton's law of universal gravitationUniform resource namePointer (computer programming)Maxima and minimaImage resolutionMoving averageMetropolitan area networkCurve fittingClique-widthMathematicsLine (geometry)Computer fileReading (process)EmailDefault (computer science)Disk read-and-write headMagneto-optical driveGastropod shellTable (information)Function (mathematics)MUDLocation-based serviceOctahedronOvalNetwork operating systemDrum memoryLink (knot theory)SimulationInterior (topology)Information managementElectronic mailing listMalware5 (number)Normed vector spaceGraph (mathematics)Term (mathematics)MIDICellular automatonTunisSimilarity (geometry)Online helpAzimuthConvex hullGraph (mathematics)System callComputer wormVaporSineMeta elementSoftware repositoryInteractive televisionGraph (mathematics)Projective planeTerm (mathematics)Keyboard shortcutFunctional (mathematics)Similarity (geometry)Computer fileMoment (mathematics)Multiplication signBitProgram slicingScripting languageDirectory serviceCivil engineeringWeb browserUtility softwareElectronic mailing listLink (knot theory)Electronic visual displayDemo (music)Integrated development environmentLine (geometry)Medical imagingCombinational logicCellular automatonoutputTable (information)Standard deviationInfinityInstance (computer science)Interface (computing)Computer configurationCorrespondence (mathematics)StapeldateiCodeModal logicQuicksortDifferent (Kate Ryan album)ThumbnailData conversionKernel (computing)Function (mathematics)LaptopRepository (publishing)Computer iconComputer animation
33:32
SummierbarkeitExecution unitLie groupEmpennageSpecial unitary groupCellular automatonMenu (computing)Wide area networkSimultaneous localization and mappingValue-added networkEmulationLiquidMUDTrigonometric functionsHill differential equationSimulationUniform resource nameMoving averageSoftware repositoryPresentation of a groupAsynchronous Transfer ModeComputer iconRepository (publishing)Medical imagingInstance (computer science)Computer animation
34:34
Maxima and minimaScripting languageCellular automatonSocial classTable (information)ArmInformation management3 (number)Server (computing)LaptopComputer iconBlogSoftware repositoryLink (knot theory)Directory serviceSeries (mathematics)Kernel (computing)EmulationUniform resource nameLevel (video gaming)Metropolitan area networkRule of inferenceNormed vector spaceSoftwareRevision controlMenu (computing)Personal area networkComa BerenicesWide area networkMalwareHookingPhysical lawElectronic meeting systemManufacturing execution systemAmsterdam Ordnance DatumGamma functionRootTap (transformer)Software engineeringMUDSpecial unitary groupExecution unitArc (geometry)Newton's law of universal gravitationBinary fileSummierbarkeitArtificial neural networkPresentation of a groupPoint cloudMultiplication signSoftware repositoryIntegrated development environmentType theoryCore dumpLaptopBlogHome pageComputer fileOrder (biology)Repository (publishing)Motion captureCodeMedical imagingComputer iconPoint (geometry)Revision controlStructural loadGreatest elementGastropod shellBitStapeldateiDirectory serviceEinbettung <Mathematik>Video gameSampling (statistics)AreaCellular automatonNumberLine (geometry)Computer animationXML
37:47
Extension (kinesiology)Slide ruleEmailLaptopData conversionExplosionLine (geometry)CodeModul <Datentyp>EmailTable (information)Process (computing)Computer fileElectronic mailing listComputing platformLink (knot theory)MiniDiscVolumenvisualisierungLaptopFile formatSpacetimeData conversionProteinData managementSlide ruleComputer animation
39:29
MUDArmSpecial unitary group3 (number)Maxima and minimaRing (mathematics)Computing platformSummierbarkeitMagneto-optical driveoutputDuality (mathematics)CAN busChord (peer-to-peer)Uniform resource nameDenial-of-service attackInclusion mapCodeGoogolWeb crawlerEmulationAreaCodeSoftware developerLaptopFunction (mathematics)EmailMiniDiscTable (information)Computing platformPoint (geometry)XMLComputer animationSource code
40:22
SummierbarkeitSpecial unitary groupEuler anglesRule of inferenceThresholding (image processing)MiniDiscWide area networkComputing platformConvex hullExecution unitMenu (computing)MathematicsElectronic meeting systemLaserUniform resource nameInfinityDuality (mathematics)MUDInformationEmulationMotion blurArc (geometry)Cache (computing)Moving averageSimultaneous localization and mappingMIDIRaw image formatHill differential equationComputing platformSpacetimeDifferent (Kate Ryan album)Form (programming)TwitterBitCASE <Informatik>Line (geometry)Right angleMiniDiscDisk read-and-write headMultiplication signLaptopTable (information)CodeStructural loadLink (knot theory)EmailCellular automatonPhysicalismGroup actionState of matterArithmetic progressionGraph (mathematics)Cross-platformPoint (geometry)CodeLoop (music)Computer animation
43:02
LaptopEmailoutputTemplate (C++)File formatInterpreter (computing)Kernel (computing)Gastropod shellAreaFood energyComputing platformProduct (business)Template (C++)Link (knot theory)Moment (mathematics)Message passingKernel (computing)Computer fileTraffic reportingMultiplication signTheoryQuicksortFunctional (mathematics)Set (mathematics)Interpreter (computing)Mixed realityMeta elementType theoryConsistencyStapeldateiComputer animation
Transcript: English(auto-generated)
00:00
Day of the conference or sad face So our first speaker for this session is Michael bright. It's going to be talking about Jupiter for everything else Thank you. So Jupiter for everything else seemed like an arrogant title to me, but at the end of the day It's just this premise that you don't actually have to be an astrophysicist to use Jupiter
00:26
many other uses and So one of them is actually a command line if I'm quite a few uses just using command line So I hope even the astrophysicists will find something interesting to take away from this talk
00:41
This isn't what I do at work either, so I'm a Brit who I did my brexit in the other sense 24 years ago I live in Grenoble in France And I work for Hewlett Packard Enterprise What I'm showing is nothing to do with my work Though when I can I do slide Jupiter in there for some things and I also run a Python user group in Grenoble
01:03
So if any of you passed by Grenoble and want to present you'll be more than welcome People will be glad to see something other than my beta talks of conference talks Okay, so I'll talk briefly about
01:21
IPython due to Jupiter themselves. Sometimes it's not that clear or the difference between them given the the big split I Look at the Duke Jupiter project and the ecosystem Which is I figure if you're going to call it Jupiter for everything else You have to look a bit at the existing stuff and there's a lot of stuff in the Jupiter universe
01:42
And then I'll get on to some of the things that I've been playing with So Jupiter for everything else Okay, so it all started out with this afternoon hack in 2001 by Fernando Perez who basically wanted Something to help
02:01
scientists in this Exploration process where you do some work individually then you share it with with colleagues When you get to a certain point you want to maybe test that on the production system and then publicize and then eventually when your work becomes established and
02:21
educate people and He wanted to start something that would help with that Exploration loop and the publication associated with that So in an afternoon, he wrote this thing 259 lines of of Python the IPython interpreter. It was just a console application
02:41
Actually, this screenshot isn't that this is the latest version IPython 5 which came out just a week or two ago But the basic idea there is the same of a text-based console Redeveloped print loop with a series of input and output cells
03:02
history and also external plotting capabilities That was the beginning and then in 2014 at SciPy They announced the big split at that point everything on this slide Existed and was called IPython
03:22
Okay, an IPython had grown into this thing where the actual interpreter was the IPython kernel or other language kernels Which were fronting? Fronted by these console or Qt console or what was then called the IPython notebook
03:42
So that was all Python and it was multi-language and it was kind of difficult to go along and say hey You know, you wanna do Haskell, Julia, R Use the IPython notebook Didn't quite make sense. So the big split was partly a naming exercise Becomes Jupiter and Jupiter because the principal languages proposed were Julia, Python and R
04:05
But as we will see there are many languages available And so now When we talk about IPython we're in fact talking about just that top left ellipse there the IPython kernel
04:20
Okay, so we have the IPython kernel or a whole host of other language kernels that can work with the front ends on the right And they are all part of the Jupiter project. Jupiter project represents everything that is language agnostic Okay, so the notebook, I guess people know a bit the notebook, but
04:42
again, it's the same concept of We didn't print loop that extended to the the browser where we now have rich media in a web page So we can have a mix of markdown cells input code cells and the output and the output itself can be
05:01
rich media being itself HTML Everything that's associated with that. So JavaScript CSS SVG. It can be hyper tanks links it can be an iframe embedding a webpage it can be an audio player to see that or even YouTube video
05:21
Personally, I mean why did I do this talk? It's because I really like the notebook concept. I like that you Develop a narrative in your notebook It's not something that you would generally use for developing code that you can do The risk is that it's very incremental and so, you know, you miss the design stage which is important, but nevertheless for
05:42
Exploratory work. It's a really nice environment and we see I've been using it actually for running demos or For developing a lab for developing a lab. It's very nice So you have a preset series of steps, even if the lab itself is not run in the notebook But it could be at least you have a way of validating your documentation of the lab
06:05
Okay, so narrative is very important important element There are many public notebooks, I have no idea how many there are Apparently there are more than 200,000 on github and github
06:21
since May last year They will render Jupyter notebooks Jupyter notebooks are actually in a JSON format. So it's not something that's obvious that github should be able to render But that is the case and there are several sites several galleries where you can look at existing
06:40
notebooks Actually, I like this last one. Not sure if I can go there. Okay Well, the network is working well I like this site because you can look at the most recent notebooks We can see yesterday there are about six new notebooks added to this repository
07:03
And we can also see there the most viewed So just to say there are many resources for viewing notebooks online These of course are just static notebooks. Not notebooks that you can run Nevertheless there are
07:23
Cloud hosted environments where you can run notebooks. I wanted to just mention Some of the big cloud providers now, so Microsoft with Azure machine learning studio Google the data lab and IBM data scientist, they're all providing online environments where Jupyter is integrated into their data science labs
07:47
I think that speaks a lot for How the community is building upon Jupyter? Otherwise, they're also another couple of cloud hosted options. Try Jupyter
08:00
And binder I think have a slide on okay first on the azure ml studio, so this is Microsoft machine learning studio environment They provided initially python as a language in that and they more recently added think in March they added are as well
08:21
It's still early days for that, but they seem to have serious plans the use of Jupyter in their environment Okay, try Jupyter is actually a rack space hosted Environment they're using a Docker demo image, which has a set of language kernels already installed
08:44
Not just languages this one actually includes the Apache spark environment I can see there are example notebooks welcome to spark with Python and Scala They're also bash Haskell Julia and Ruby kernels there
09:02
That's just an ephemeral instance You go to try Jupyter.org you will get this dashboard and These notebooks are these languages available to play with I will talk later about binder And how I've been using that which also provides ephemeral instances
09:22
Okay, so just to talk about the Jupyter project itself some of the components of that So the notebook runs in the browser It has its notebook server, which is written in Python itself and has many HTML JavaScript elements associated I'm behind it
09:41
We have a language kernel whether it be Python Ruby or whatever and the project includes so language kernels widgets so things like ability of drop-down boxes menus and stuff within the notebook and language kernels
10:01
So we look at the kernels So there are about 50 kernels available this list probably isn't up to date I should say though your mileage might vary I've tried a few well. I'm running windows here So it's my own fault, but some of those things can be quite tricky to install some are
10:20
incredibly easy to install I Think I have installed them working the meta kernel bash. I have some Callisto prologue Callisto scheme conch which is a Sort of Unix Python shell that was announced at PyCon a few weeks ago What else Python of course?
10:44
But there's there really are a lot of languages squared there Okay extensions There are a lot of notebook extensions. I'm not going to go into detail about those I'll just mention some that I find of particular interest So rise
11:00
I'm running rise now. So this presentation is actually running in a Jupyter notebook and there is this rise extension which allows you to run a Reveal.js presentation from within the notebook There's also nb presenter. I only discovered this last week. That's that's what I love about Jupyter
11:23
You know your presentations already and then you discover another amazing extension So nb presenter is produced by the guys who produce anaconda Continuum IO And it's another way of doing Slideshows within Jupyter. I've installed that but I haven't dared use it use it for the conference
11:45
I was too frightened of breaking my existing slideshow nb grader, I mean there are hundreds of extensions. These are just ones I find of particular interest nb grader is something that can be used in classrooms
12:02
There's a thing called JupyterHub which allows you to have a multi-user server serving up Notebooks different users in the classroom setting and nb grader is an extension which allows you to set an assignment cells will be defined as automatically graded or manually graded
12:22
How many points are associated in a particular part of an assignment That's quite an interesting development Okay, if we look at the ecosystem and some of the future projects So some of the more interesting ones are spark magic
12:43
So we've seen on the trijupyter. There's already a spark integrated into that image, but that is It's spark integrated into the image in general when you want to do some real data analysis with spark You're going to want to have something A real spark back end externally. So spark magic is a project to allow that a
13:05
Standard way to integrate into external Spark environments so I can imagine with as your IML for example, they will no doubt use that to integrate with their own spark Declarative widgets I mentioned widgets top-down menus and so on today we can use IPython widgets
13:24
But that's Python specific. It's in the IPython kernel Declarative widgets will allow to define menus drop-down boxes and so on within the HTML Which can then be more easily? integrated in the different language kernels
13:41
And dashboards will show you in just a moment Content management. There's a lot of work on looking at how we can Import a notebook as a module. Okay today a Notebook is standalone. That's what you get but there's work looking at how to
14:00
bundle a notebook with other dependent files Import notebooks as a module or as a cookbook And also do things like tables of contents There's also the kernel gateway project which will basically allow Non-notebook clients are basically any web front-end, for example
14:23
to integrate elements of Jupiter So I have an example of something similar in a moment where O'Reilly Integrated cells of a notebook into a blog post Okay, so I would just show one of those incubator projects this one is already
14:46
Available you can pip install it. I've not actually done that yet. I think it's in its 0.6 release something like that and the idea is Being able to move away from that linear notebook
15:01
concept And to be able to arrange your cells into a grid as you would want to do something Which you want to use as a dashboard This isn't an incubating project as such but it's up and coming. So I think it was last week
15:20
It was announced as a pre alpha Jupiter lab Jupiter lab will be the new interface. So The interface I showed for tri-jupiter I was showing the dashboard in the future This Jupiter lab will be the standard interface where we'll have the same
15:40
Possibility of having a list of files and folders Opening a terminal opening notebooks opening notebooks in different tabs. You can see that it's quite a dynamic environment It's only it's a pip install away it seems pretty stable Actually seems totally stable from I've seen but it is pre alpha so you don't have the same functionality yet
16:07
That's pretty nice Personally as I've done Demos using the notebook. I like running demos in the notebook and everything is controlled But it's nice as well to be able to pop out into a terminal and I like it's the Jupiter lab environment
16:25
Are we able to have my my notebook running and in the same window a terminal? To see what's going on Okay, I was just given idea about things that happening in the Jupiter project
16:41
There's a lot of interesting stuff, especially on the user interface side of things I'll just mention that there are a lot of Projects using Jupiter for blogging there have been examples of nature and scientific American
17:00
Doing blog posts or people doing complete books. I think it's Python for data analysis is available As a complete set of Python notebooks Jupiter notebooks And in education, there are various MOOCs online courses that are using Jupiter online
17:23
Okay, and I just mentioned that O'Reilly blog article actually I've been having trouble running it too recently But the idea is it's just a standard blog Looks like any other blog page on the O'Reilly site But within the article they have these
17:42
Blackened out cells on the right and you can't see it, but there's actually a run button bottom, right? And so as you read the blog article you press run that's going to launch actually a docker container running As you can note notebook instance and you can actually run that off sleep modify the code as you go
18:04
Okay, so if I look at what I've been doing with Jupiter everything else Of course with Jupiter as I say you have this rich media through Through markdown or through code you can inject HTML CSS
18:22
JavaScript SVG into your notebook very rich environment I've been looking at the use of the bash kernel for command-line work. Why not? And then I wanted to see well I Like the idea like the idea of having a
18:43
Unix shell As a first-class citizen itself, you know, why shouldn't you just be able to do command-line stuff as well in the notebook? But why not supplement it as well with with your affix? I've been playing with with that a bit You could say why of course because you could just do Python and then escape out into bash
19:03
Well, there are several reasons for one escaping out into bash Can be slow you're launching a sub process and there's no persistence between Cells when you do that But the the other reason was I felt what I want to do a tutorial for example, which is command-line stuff I
19:25
want it to look just like the shell and With the graphics as well, I don't want to be breaking out into magics. Okay I've been looking at So I mentioned the right extension. I'm running here. I've been looking at publishing live blog posts. In fact
19:46
It's not quite that people are using Jupiter to create blog posts. That's not what I've been doing I've been blogging and then linking to a server binder where you can run a live instance of a tutorial and I find that pretty cool
20:02
And then the other things I've been doing it's been Out at work. So I work in telecom we manage About six open stack platforms Open stack is a cloud management platform And each of those systems is between probably about eight and thirty physical nodes. So there's a lot that can go wrong
20:25
so I just I've done some just simple stuff and a cron just monitoring a bit those systems I'm not saying that is how you should manage those systems It was just some play quick and dirty and it decided to extend that and thought well, why not use the notebook?
20:44
I saw people were talking in the mailing list about using facility called NP convert So I gave that a shot so look at that as well Okay, I won't talk much about the web technologies as such I'll look at it within
21:05
My command line stuff just one thing I'll say is I mean you have access To the browser. So yeah, you have enough rope to hang yourself by I won't hang myself with this normally, but just here's an example where I can I can't see very well
21:21
Change the theme of my presentation That's just a bit of JavaScript a bit of CSS HTML and you have full access But you can do lots of things that will Break your notebook if you if you play too much with CSS Okay, so looking at what I've been doing with the command line
21:43
There are actually two bash kernels available and I went for this one The Egypto account Callisto, they've got a set of meta kernels And I like these basically because well, first of all, I could get it working on my windows with just one line fix
22:03
But also because they have a family of meta kernels. The idea was There are magic's and IPython but those are specific to the IPython kernel and here there are a set of magics That are available for all their kernels, so I mentioned that I have the scheme the prologue another thing called coconut
22:22
I think a set of kernels have the same common magic. So I like that and it's also on the active element So why well, that's my boat As I said, I started out actually doing some demos I was actually doing some docker demos at work doing presentations then demo
22:44
I started out doing them in Python escaping into bash very nice Started doing them in command line and then also I ran a docker build lab in February And so I developed the whole lab in a notebook and that was that was pretty nice
23:01
So I was I was pretty late preparing and at times when you've changed things, you know, make sure everything works It's nice to be able to do just run all And your whole lab just runs from start to finish and then you can see if there are any problems
23:20
Okay, and I wanted to look at Can I make bashes to first-class notebook citizen adding some magics? Okay And as we've seen a moment I looked at some command line tutorials, which I blogged about and that you can run live on binder
23:42
So let me just Just go to an example notebook with some of the things I've been playing around with in bash Hopefully this is stable wasn't quite the case yesterday Okay, so I just wanted to play around with the different things HTML JavaScript and so on
24:06
See how I could make some of those things available in bash These first examples might seem like a waste of time But just showing you can do it you can echo some HTML into this bash function
24:21
I created and it will actually be rendered as HTML So the principle was not to use the magics which I could have used But to be able to really stay in the bash paradigm Okay simulates I can create an HTML table
24:45
Okay a more practical example so a work Okay This is an example of running a the open stack command-line client and seeing a couple of VMS that are running So I captured that although I'm connected to work. I wasn't going to depend on the
25:01
network connection So I can echo that same text pipe it through this thing HTML table and display it as HTML Okay, nothing amazing Then if I define a function highlight say I want to highlight all the VMS or actually all of them are active there
25:21
Then I can do that same same pipeline deciding to To highlight the active stuff or whatever Okay, so it's not rocket science But the there could be uses for that Or
25:41
Similarly just doing some basic stuff Okay, curious Okay, so this is the function definition so I just defined a function HTML find just be able to
26:02
Output the results of that as an HTML table again. It's nothing nothing amazing itself Okay One of the magics available in the meta kernel bash This thing percent percent dot. It's basically graph view
26:21
Graphics, so it allows you this is line magic where you can just create a a graph on the fly So I wanted to I wanted to create a a bash function to be able to do the same thing okay, this is really just
26:42
Playing seeing how we can make something which is essentially command-line make it more visible if you want to run a tutorial Some other examples and this one Do I display it? Okay
27:01
In this one. All I'm doing here is just creating a file hierarchy and then I created a function to the display that graphically Which is broken, okay. Never mind No, it isn't broken. I'm broken Try again, okay
27:22
so nothing amazing just Just trying to mix the paradigms again can do just basic JavaScript do a pop-up Amazing. I'll skip the Python one. Let's go By being Python into Python from back and bash is a bit weird. I don't have a use case for that one
27:43
I did An Example of just doing some graphics with d3 gs Not very pretty but I just wanted to push see what I could do This is fairly Empty example, really, but I'm actually calling lightning lightning is a graphics library with publicly hosted server
28:07
So the content of this is just Not something useful we're afraid but it was just to show the example Similarly, I did the same sort of thing with bokeh
28:21
Okay The intention would be to create a minimal API around a bash function to invoke bokeh So I've done some stuff like converting CSV files into HTML instead of thing I'd like to do the same thing of doing CSV
28:43
to bokeh and then maybe with a Minus by chart minus bar chart this sort of option. Okay, I literally copied some Some code from the other bash kernel Done by someone from the Jupiter team just to be able to do image display and all my functions
29:05
I tried to make them work in different ways of display of a file name display on the standard input Okay, I'm gonna skip the SVG is just some magic. Okay, that occurred
29:26
I Mentioned so I wrote some Python So here it's just in the cell just to show you That would do conversion from CSV file to HTML See what I could do with that. So
29:42
See, I'm just using a DU taking the first five lines just see what we get Doing that the minus H human option then sorting that. Okay, so we got the largest files first 10 megabytes for demos directory
30:01
Now I'm going to pipe it into that Python script Okay, and this will output HTML. I'm doing ahead just take the beginning so you can see that it constructs The HTML now did the same thing but piping it into the HTML function
30:21
Okay, so, you know, that's very much in the UNIX command-line spirit of piping stuff into into different functions Okay, I'm gonna skip the rest
30:42
Okay, there was one other thing I did because I started to do just you know Like lists of files and this sort of thing. I thought it could be interesting It reminded me of project called XML term and days of my youth. Well, actually I wasn't that young but back in 2000 this was like pre Firefox
31:04
Motsila was the browser and some guy added this capability called XML term and Well, basically that allowed you within the browser to have a command-line interface and it would do things like thumbnails and so on And you could it would actually have utilities which communicate XML and I believe to XML at the time
31:24
I thought this was going to be really great but unfortunately Motsila was a was a big thing and it's a good thing that it got replaced by a Firefox and so it got It got cold nothing ever became of it. I was surprised a few years ago to come across a thing called graph term
31:43
Actually developed by the same guy so this time it was all in Python So I was just wondering well, can I do something similar? Okay, so I created a function XLS. I see that's a tribute to XML term because he had a function called XLS
32:02
And so the idea was It's a bit slow because there's a lot of rendering but it will look Different files in the directory and it will render them in different ways. So folder icons and folder Jupyter icon for notebooks The CSV it's rendered as an HTML table
32:24
Docker file the nice whale Okay So I had fun with that Personally what I find more interesting is binder So binder is a resource that for moment is free
32:43
So I shouldn't be talking to you about it because everyone runs out and uses it and maybe it won't be free anymore This is the interface to binder Basically, you give it the URL of a github repository. It will look to see if there's a docker file And it will build a docker image corresponding
33:02
If there isn't then it will look for a pip requirements file or a conda YAML file, but basically it will on the fly. Well, it will build and a docker image and Then you or anyone else if you share the link with them Will be able to to run instances of that environment
33:23
So you have instant runnable environments that have all the dependencies necessary for running a jupyter notebook So, let me see Here's an example. Actually this example is the dashboard when you give it a repository This is actually my github repo which has this
33:43
presentation in it so if I If I click on launch here, which is rather daring mid conference It will actually launch an instance of The docker image that I'd already built from from my github repo
34:04
Okay, so in the dashboard I can actually go
34:21
Okay, I was just hoping for one thing that I would have the icon of the rise presentation Okay, there you go. This is the icon to go into the slideshow mode Okay, and that Is my presentation at this time running out in the cloud on binder so I find binder really cool
34:44
I'll kill that But more interesting for me So I created a github repo with some command-line tutorials And I linked to that
35:02
So you go to that github repo basically when you build an image on binder they They propose that you put this launch binder icon on there and if you click on that It will take you in it to an image running that repository. So again, this is
35:27
Live in the cloud So I there I just initially as I blog posted and I thought well I'll do a load to these before euro Python great So I thought I'll start at the bottom and I did an LS tutorial. Yes
35:42
And it took me another two months to get around to doing anything else, but thankfully I did a week or so ago I added a few a few also especially for you. I added a pearl six one-liner tutorial It's just a shell it's an idea but now something more interesting So here this is an index page the different tutorials. I created and one of them. I like the idea quite a bit
36:07
so I did a TCP dump tutorial it is pretty short, but you know just get the idea So I always start off by Just doing a check of what environment we're running
36:23
And so running the first cell we can see that we're running a Debian Debian Jesse this is running on on binder What bash version we're running what TCP dump version we're running now, we're not going to be capturing with TCP dump
36:40
Out there in the cloud. We probably could do if I wanted to I'd have to do things in the dockerfile to enable that But it's more interesting in a tutorial anyway to have a pre-captured file and go through that So I describe in the markdown how you would capture your own packets
37:02
And then we see I have three examples of packet capture files How we can rerun the code To look at those so let's say I want to look at just the first two examples I'll put a minus C2 in there. I rerun we can see that no such file a director
37:21
Yeah, well if you try and type at conferences this happens You know okay if I put that in the right order you can see okay. She was just the first two so the point I'm trying to say is you know this is a Live tutorial I linked to it for my blog and anyone can run that and play with something like TCP dump in the notebook
37:43
environment I find that I Like that concept I really do Okay So I've talked about rise for doing slideshows, I won't go into more detail right here
38:02
Something else I did then as I mentioned we manage about six open stack platforms and four of them I Basically I wanted to look at managing them see how they're doing for disk space and so on It's actually after seeing someone talk about nbconvert on the jupyter mailing list
38:24
Thought I'll give that a try then be convert it can execute a notebook And it can obviously convert to another format so you can call it from cron so I have a cron job that nightly Runs a notebook under cron
38:41
Converts it to HTML, and then I send that HTML to me as email now There were several problems along the way. I thought great. Okay. I'm now supplemented with graphics except that Email readers will render HTML, but they won't render graphics unless it's actually a
39:04
Link to an actual external graphics file I Okay, well I'll try CSS so I developed some CSS stuff and ah they don't they won't do the animation of CSS either Same thing with obviously the JavaScript and even SVG so okay never mind
39:25
So I looked at what I could do and obviously HTML tables So if I look at okay here we have a live example. I'm actually gonna clear all the output and rerun this
39:43
The idea is okay I developed this in the notebook then periodically abstracted stuff out so that the there would be Relative little code showing in the notebook as this will end up in an email Let me just Initially I did these basically one per platform. I know to create HTML table showing the status
40:06
this is Physical nodes they respond to ping this is the VMS on the active or not This is checking where the ports are open and this is checking some of the API endpoints And then I was also getting some disc uses stuff
40:21
that one I've actually abandoned because The end of the day You know you want a dashboard you want something that will give you a heads up is everything okay, please So I basically refactored it. I'm going to run all that now. It's a bit slow
40:41
Okay, this is connected to our lab in Grenoble I Mean this is just showing the progress is pinging physical nodes, and it will go through the four platforms
41:02
At one point when I wanted to do an initial status Table at the top for the multiple platforms I thought why yeah with the notebook and do some JavaScript and just insert that up in the top of the notebook But I forgot of course that that won't be rendered in my HTML email, so I just had to refactor the code
41:23
Each time I got a platform status. I've returned Some status codes to me and the HTML and then afterwards Once all the status have been done, I'm just coming through to the fourth platform now I have another cell where I
41:40
Construct a table of the overall status the different platforms, and then I loop through the different platforms Printing the HTML detail of what happened, okay, just let that finish
42:01
Okay, it's very quickly So I get this platform status summary and of course It's some HTML so I can link link through so I look at the second line there Platform platform called nfe5 what I've done is Shown on the right the status of the ports and so on everything's fine
42:21
But we can see here the disk usage, but this platform has about 30 odd physical nodes, and this is the node which has the worst case disk usage If I link down to that platform see those details on the disk space So I've ordered the disk space of lots of different nodes
42:44
So this one was the worst one and here I built a bar graph showing the trend over the last 10 days or so So at the end of the day this space is high as a problem What's worse is if it's going up like 10% a day and so on Okay, so I just wanted to show that as an example of how we can use nbconvert
43:06
to do status reports I'm not saying this is the way I should monitor our platforms, but it was more of an experiment with Jupyter Okay, and this is the sort of command that I used another little gotcha with it was with Gmail in particular
43:26
It would clip the message saying I have to go on another link Basically the standard template used there's too much CSS and so on So using template basic you can cut down the the HTML file size
43:45
Okay, so to finish off I'd like to go further with this using JupyterHub and nbgrader trying to do some online It's a classroom type labs
44:02
Experiment more than the meta kernel batch stuff. I'd like to Have a more sort of consistent API set of functions that can use so I can do things like bokeh graphics and so on but from some real data, I Want to do some pull requests against the meta kernel bash. I'm out on the limb at the moment
44:23
Propose this stuff outside the Python community will actually got rejected for LinuxCon Europe. Never mind But that's fine There's this conch kernel I mentioned I'd like to take a look at that though, I suspect It will be a weird mix of command-line and Python
44:42
it's probably not what I'm looking for and also look at there's a C++ interpreter kernel, which has a So a kernel for the notebook, okay, that's that's it. I think I ran over