A Jupyter Enhancement Proposal Story
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 132 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/44892 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 2018113 / 132
2
3
7
8
10
14
15
19
22
27
29
30
31
34
35
41
44
54
55
56
58
59
61
66
74
77
78
80
81
85
87
91
93
96
98
103
104
105
109
110
111
113
115
116
118
120
121
122
123
125
127
128
129
130
131
132
00:00
MathematicsMoving averageComputer-generated imageryAxiom of choiceExecution unitProjective planeProbability density functionScripting languageObject (grammar)System callLaptopSeries (mathematics)Computer configurationSheaf (mathematics)GradientBlogTouchscreenWebsiteSelf-organizationMedical imagingNumberJames Waddell Alexander IIPiCellular automatonPoint (geometry)File formatMultiplication signOpen setDifferent (Kate Ryan album)Data conversionLattice (order)MathematicsIntegrated development environmentPhysical systemCodeGraph coloringWritingLie groupLink (knot theory)TwitterSuite (music)CodeWordSoftwareAxiom of choiceHeegaard splittingFunction (mathematics)Revision controlSoftware development kitNeuroinformatikMathematicianBlock (periodic table)
06:28
Modul <Datentyp>Template (C++)Demo (music)LaptopConvex hullMetadataExecution unitDirectory serviceOnline helpMeta elementBlock (periodic table)EmailCodeSoftwareMixed realityDemo (music)Multiplication signBoss CorporationPrototypeRepository (publishing)Digital photographyComputer programmingLaptopSocial classIntegrated development environmentMereologyMathematical analysisComputer fileFunction (mathematics)TouchscreenWeb browserMoment (mathematics)File formatTraffic reportingTime travelDifferent (Kate Ryan album)Medical imagingType theoryWebsiteTable (information)Computer configurationTemplate (C++)Source codeSpacetimePlanningSuite (music)BitCase moddingCartesian coordinate systemCountingGastropod shellGoodness of fitProgramming languageInterior (topology)Software developerContent (media)Scripting languageComputer animationEngineering drawing
12:56
Axiom of choiceComputer programLibrary (computing)Block (periodic table)Data modelRegulärer Ausdruck <Textverarbeitung>CodeProgrammschleifeTouchscreenLaptopTemplate (C++)InformationComputer animation
13:15
Directory serviceLaptopOnline helpExecution unitInheritance (object-oriented programming)Error messageAirfoilTemplate (C++)LaptopFunction (mathematics)MathematicsInformationWritingWeb browserLine (geometry)Data conversionComputer animation
14:32
Axiom of choiceComputer programDecision theoryComputer-generated imageryBlock (periodic table)Data modelCodeComputer wormTemplate (C++)Computer animation
14:55
Directory serviceError messageLaptopOnline helpFluid staticsComputer-generated imageryGamma functionDifferent (Kate Ryan album)Medical imagingTemplate (C++)Musical ensembleComputer animation
15:36
Axiom of choiceComputer programMIDIComputer-generated imageryBlock (periodic table)Normed vector spaceLine (geometry)InformationCodeEmailTraffic reportingAddress spaceComputer animation
16:05
Demo (music)LaptopPrice indexPauli exclusion principleLaptopMultiplication signFeedbackTemplate (C++)1 (number)Moment (mathematics)QuicksortCASE <Informatik>TwitterTerm (mathematics)Series (mathematics)Metropolitan area networkOrder (biology)Process (computing)Water vaporMereologyOpen setDirectory serviceMedical imagingBitMetadataMathematicsInterface (computing)PrototypeInformationDefault (computer science)Procedural programmingComputer fileFile formatRepository (publishing)Self-organizationComputer animation
Transcript: English(auto-generated)
00:02
Thank you. So Tanya sent her apologies. She was double-booked, so she couldn't come. So my name is Hanyeri Silva. Most of the time online, I'm using R-G-A-I-A-C-S as my username. So if you're looking to find me online, that's my username.
00:25
And Tanya, that's her contact if you want to find her online. She's pretty active on Twitter. So how does a triple proposal kind of start? It was last year on this same city in Edinburgh.
00:42
So there is a project from Arizona 2020. That's a big grand project from the European Union. And one project under that umbrella is the Open Enduring Kit. So the idea for this project is to work on infrastructure for researchers. And one of the goals is to strengthen
01:05
virtually researched environments. So for example, Jupyter's notebooks and all the Jupyter ecosystems. And last year, as I was saying, we had a meeting that was the computational mathematicians with Jupyter and at the International Center for mathematical science. So the Institute's
01:29
software sustainability Institute where I work, we have some fellows and one of the fellows is Alexander Konal Valov, and he's one of the co-investigators for the Open Enduring Kit project.
01:44
So he kindly invited me to join the team for this one week half talks, half hockey day, which was quite nice. So during all the conversation that people was having, we started talking about
02:04
how to develop the lessons because most of the people there were interested in that. So at some point it was like, we want to have lessons in a website that looks nice. Not exactly as nice as this one because it's an old one, but I got involved in Software Carpentry.
02:23
That's a non-profit organization that teaches people in academia how to code in Python and version control that software. And this is a screenshot for how that lessons was looking like in 2014. What you can see on the screen is that they have a title for this lesson.
02:44
So it's Making Choices, and they have a little below a series of objectives for that lesson. So for example, readers suppose you know how to create a single image made out of color blocks at the end of that lesson. So it's a little not too long lesson, but you can see some quotes at the first
03:06
section. So image grades, you can see some Python quotes and some cells. So the idea that was discussing on the workshop was like, it would be really nice to end up with some lessons that has this kind of look, a little more professional, and with some colors that make readers find things easily.
03:27
But that you could write the lesson in Jupyter's notebook because if there's some code and some output, you don't want to copy and paste every time that you change, especially if you have 100 cells on your lesson. So we were looking on Jupyter. So this is the same
03:46
lesson that doesn't have the title, but you can see that the objectives are the same and the same Python quotes. So this is Jupyter. At that time, 2014, we didn't use the word Jupyter. It was
04:01
Python notebooks at that time. So it's the same notebook that was generated in that previous image. But just on Jupyter. And it was like, so this is how you have on Jupyter, and this is how you're writing, which most people on academia that use that too is already familiar with.
04:23
But this, when you're trying to export this, you can choose someone that's going to read the lesson. You can say, oh, I can use nbview. That's a nice tool. I just need to send a link, and this is how you're going to see on nbview. Which is okay, but as you see, you still have like the numbers of the cells here,
04:45
which as you see, start on six on this example. And why? They really can think, why start on six and not one if they start of the lesson? And there is no color, so it can be a little hard for people to follow the lesson because it doesn't make
05:01
very highlighting things. And it was keeping these discussions like, okay, so what you can do? And if you ever use Jupyter notebook, you might have come across with the option to download your notebook in different formats, so you can download the notebook itself.
05:23
But you can also download as a Python script. That was something that I mentioned yesterday on the beginner's day for someone, that person was asking like, if I want to get just a script from the notebook, so you can just go and download. You can download as HTML file or markdown or
05:43
restructures text or LaTeX or get a PDF via LaTeX. So there's a couple of options, and it was like, okay, so let's see how end up if you decide to download as an HTML, because you can upload the HTML to many websites, and they just render. And it looks exactly as nbview, because that's basically what nbview was doing. And it's like, okay,
06:05
we still have this kind of same problem. We still want to remove some things and add some things. So keep on that discussion and not think that people kind of raise. It was like, sometimes you have one lesson,
06:20
but you want to split the lesson in sublessons. So maybe once you split in like day one and day two, because it's a two days workshop or morning or afternoon, and you might have extra contents, not just Jupyter notebooks. You might have Python script, because you explain for people how they're going to use scripts, or
06:43
you might have some file formats that you want to download, because you have some chemistry files that people need to read and so on, and you want to make it easy for people to download all these files, so you want to organize all the lessons in some format.
07:02
So this screenshot was something that Tanya ended up working on during that week, one year and a half ago, here in Edinburgh, and it's kind of a nice prototype. You can actually use it. But still, we were discussing how we can use better Jupyter notebooks, and
07:24
the main thing that was like, why do you want to use Jupyter notebooks? It's because the environment is familiar to authors, and it's part, at this moment, 40 program language, so it's not just Python. If someone is writing a lesson in R,
07:43
they can use the same infrastructure. If someone is writing a lesson in C, the same thing, they just can build on top of what everyone else already contributed. What pieces of the pipeline we kind of built on top? So there is nbconvert. Can you raise your hand if you know
08:05
what nbconvert is? So that's half of the room. So nbconvert is what's behind all that options that I was showing before on Jupyter notebook, that you can download the notebook in different formats, and
08:21
there are other things, any site generator like Sphinx, Hugo, or Jekyll. So that was the two pieces of work that you need to build on top. And Tanya was, again, working on other ideas, mixing Jupyter notebooks and Jekyll, so that's one of the Git repositories.
08:44
And fast-forwarding a little bit, this is the Travis, sorry, the time machine from Doctor Who, so you are in England, so I want to put this image.
09:01
I ended up in Europe item last year, so that's the photo of some discussions, not on the discussion, but some of the people was close by. So it was Thomas, Thomas is in the back. He and I was in one of the helping desk, and someone showed up saying, so
09:23
can you help me with something on Jupyter notebooks? And it's like yes, so what do you need? And that attendee was like, so I have to create some reports, but every time I need to put some logo on my reports, because my boss needs those parts to be archiving something.
09:44
And all the analysis for the reports are on Jupyter notebooks, but at the end I just want to like include the logo, and I want to save time because I don't want to keep doing that every time. At the end it was like, because at the moment I
10:02
generate the LaTeX code, and then I need to edit the LaTeX code. So I don't want you keeping editing the files every time, because as you might know, as a software developer, if you need to keep doing things every time again, it's a problem, so that's why you want to automate.
10:21
And at that time on the table I was like, yeah, you could like create a LaTeX class and try to avoid to do most of the editing, and Thomas was like, oh no, but you can have custom templates for nbconvert, and I was like, oh yeah, that's true. So,
10:40
for, so that was like, it's funny because I never, I was using that option a long time ago, but I kind of forgot what was, so this is demo time, just to give you an idea of how nbconvert and the custom templates work.
11:01
So we, so I'm going to do this on the shell. It's okay for people on the audience to read. So there's a few files here. The 04-con.ipynb, it's just a Jupyter notebook,
11:25
so if I do last on that file, it's just a JSON format, if you ever open that file, it's just a plain text, so there's nothing extra special on that one.
11:42
And there is a template here, this full template that I basically just download from the source code, from the node nbconvert, so it's exactly the same file. And they use Jinja too, so if you ever
12:01
edit any of the Jinja files through templates, so you're going to be familiar with this instructions, extends, so it's just saying that's extending one previous template, and there's blockers and so on, and not going too deep on this one, but it can create, can use that template to generate the HTML file from the IPython notebook, so I can type
12:29
nbconvert, and I can pass the file, I can use template, and I can pass a file, so foo tpl, and I can pass the file name, and
12:44
it's going to generate saying that it converts, so I'm just going to switch the screen to open on the browser. So it's this one, yeah, so that's the output.
13:01
I just refresh so you can see, and it's pretty similar of one of the screens that I showed you before on the Jupyter notebook, but if I want you to say, oh, I want you to add some extra information, I can build it on top of that template, so I have a person A here, and I'm just going to show you a diff,
13:22
so you can see what I make the change, so foo 10p, and because 10p, 10p, so I just add this line, so it's just saying europy 2018, so it's just before all the notebook information,
13:44
and I can use jupyter again, and be convert template, and now I'm going to change the template to custom A dot tpl, and I want
14:02
output as 04 custom A, and the notebook itself, so it's, sorry, there's a typo here, and it's using now the template, my custom template, and
14:28
converting the notebook, so it's writing, and I'm going to open on the browser, so this is the new one, just refreshing, so be sure that's going there, so there is our europy 2018
14:41
here on the top, so as any minimal example, I can do something, I can customize my template, and then I want to do something more interesting, just like not just a text, I can maybe add an image, so that's the second one, so the difference between
15:05
the foo and the custom B, so I just add an image here, so you can see here, so just an HTML tag with the image, and you can
15:23
convert using the same command, just changing the template here, this one, and I'm going to switch to the other one, so then I have the logo of the euro Python
15:42
2018 here, so the idea is like, if I can do this with like just changing one line of code, I might be able to add any other kind of information that my employee might want to have from the report, the load, the address, any email or anything, so that's what
16:01
Thomas had suggested for her to use this feature from any big convert, and it was like, this looks very good, but it's kind of, the users are just accessing the Jupyter notebook and see some at the top of the iceberg, and
16:21
that's just 10%, the 19% of everything is under the water, and no one's kind of seen, that was the case of that attending, and the same attending that was like yesterday, on the beginner's day, so like, maybe you can see if you can prove that, and they make that more visible in terms of templates and so on, and that's what I end up
16:46
talking with Thomas on last year, and it's like, what I can do, and he was like, maybe you can send a proposal, and I was like, okay, I know that like, Python has proposals, they have their paps, and I was already kind of familiar,
17:01
I read a few of them, especially PAP 8, that's about coding formats, it's like, okay, Thomas told me that there is a Jupyter and Last Man Proposals, so I started looking online, and it's like, okay, there's a git repository, that's from the Jupyter organization, and it's called Last Man Proposals, and it's like,
17:23
reading the readme, and it's like, yeah, that's the procedure, if you want to suggest something, you can copy one of your files, create one directory, open a pull request, so that's what I decided to do, I decided to open a pull request, saying like, can you add templates as metadata,
17:41
and this is my Last Man Proposals for this pull request, and you can see that Thomas had a comment just left to me, so my idea was, because I want to reuse the same template over and over again, every time that they're going to export this, it's supposed to be saved somewhere, so that's why you need to be something
18:02
on the notebook format, not just the Jupyter interface, so that's why I ended up sending the proposal, and there was a few comments below, and I included like, an image, a screenshot, this is how I envisioned the idea, you can go to the Jupyter
18:20
interface, and you can say, this is, I want to set a template, that's my template, custom template that I want to use, and later, you can just go and download that. At the end, we had some discussions on the issue, but I ended up being too busy to just push and get like a kind of prototype, and explore more the ideas, and if you propose something, and you're really not able to take
18:45
forward with lots of time, it kind of kept open, people was very welcome from everyone from the Jupyter community, sending feedbacks and so on, so it's kind of my fault to not have too much time, but that was what I want to share with you today,
19:04
I want to thank Special Thomas for all their contributions and push me to share this, and if you have questions, I'm open to having questions now, if you want to make questions later on Twitter or anything, that's my ID and Tanya's ID. Thank you very much for listening.
19:26
Thanks, are there any questions? No? Well, what kind of, you said you ran out of time to sort of fully implement it, how big of a job do you think it is
19:42
to implement one of these proposals? The main challenge was just like learning how the insides of like how nbconvert works and the order that it reads files,
20:01
because the challenge is like if I start any, not the name of the template that I want to use, it's inside the notebook, but the nbconvert, he started doing some things before open the notebook, so I kind of was like, I'm
20:21
changing this part, it's like not super trivial, you kind of need to open the notebook, and at that moment you kind of passed that, and later you can say, okay, I have that piece of information, and now I need to go back and change the default one. So that was the most like time consuming that I didn't have time to devote too much.
20:45
Is it all written in Python, the nbconvert? Yeah, the nbconvert is in Python, it's fully Python. The templates, they have a series of templates, and those templates are in Jinja too. Okay, yeah, that's your Python. And there's the Jupyter notebook, is that fully Python, or are there other bits? No, Jupyter notebook,
21:05
there's lots of JavaScript using the interface, yeah. Makes sense, yeah. Anyone, any questions before we wrap up? Okay, well, can we thank you again? Thank you for listening. Enjoy.