MOOC Cubes and Clouds - Cloud Native Open Data Sciences for Earth Observation
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 156 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/68501 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
FOSS4G Europe 2024 Tartu82 / 156
6
33
35
53
55
59
61
67
70
87
97
99
102
103
104
105
107
111
121
122
123
124
125
126
127
128
134
144
150
151
155
00:00
CubeComputing platformThermodynamischer ProzessElectric currentTheoryLibrary catalogFile formatGoodness of fitCubeElectric generatorCircleSelf-organizationGroup actionState observerMassCloud computingE-learningCASE <Informatik>Open setTheoryFile formatMereologyFormal languagePoint (geometry)Library catalogWeb pageDimensional analysisBitService (economics)AreaLevel (video gaming)Thermodynamischer ProzessCategory of beingProcess (computing)Internet service providerVapor barrierData structureTerm (mathematics)Direction (geometry)LaptopMathematical analysisReal number2 (number)VirtualizationComputer fileClient (computing)Operator (mathematics)Server (computing)Thermal conductivityMusical ensembleMessage passingProjective planeLecture/ConferenceComputer animation
06:03
Thermodynamischer ProzessTheoryComputing platformCubeLibrary catalogFile formatContent (media)VideoconferencingStandard deviationInternet service providerYouTubeCoding theoryComputing platformWeb browserKey (cryptography)Game theoryCartesian coordinate systemPoint (geometry)Axiom of choiceLaptopCubePixelComputer programmingDifferent (Kate Ryan album)Cloud computingBinary codeSoftware developerSoftwareMereologyForm (programming)Stack (abstract data type)Field (computer science)Projective planeIntegrated development environmentResultantOpen setProgramming languageTelecommunicationVideoconferencingOpen sourceContent (media)Web browserLibrary catalogDirection (geometry)Computer architectureFormal languageLevel (video gaming)SpacetimeMultiplication signGUI widgetCentralizer and normalizerView (database)Computing platformExpert systemAreaSeries (mathematics)Electronic program guideE-learningMessage passingYouTubeReal numberSlide ruleRight angleLecture/ConferenceComputer animation
12:02
Content (media)VideoconferencingLibrary catalogComputing platformLanding pageOpen setPublic key certificateWeb browserStudent's t-testLanding pageCASE <Informatik>Multiplication signDirection (geometry)Link (knot theory)Point (geometry)Descriptive statisticsInformationUniverse (mathematics)Field (computer science)CubeComputer animationLecture/Conference
12:52
Integrated development environmentInteractive televisionVideoconferencingContent (media)Data storage deviceComputing platformDiagramBlock (periodic table)BuildingDrag (physics)Component-based software engineeringInformationLink (knot theory)View (database)AreaSeries (mathematics)Plot (narrative)Computing platformTheoryDigital filterKernel (computing)Computer fileLibrary catalogTemporal logicThermodynamischer ProzessType theoryWeb browserTexture mappingCubeCovering spaceOpen setCubeLibrary catalogWeb browserCovering spacePatch (Unix)AreaMathematical analysisValidity (statistics)Level (video gaming)Multiplication signComputing platformVolume (thermodynamics)PixelIntegrated development environmentBinary codeResultantComputer programmingDifferent (Kate Ryan album)Stack (abstract data type)Interactive televisionE-learningComputer animation
15:36
CubeStatisticsIndependence (probability theory)Open setWeb browserComa BerenicesThermodynamischer ProzessPublic key certificateProof theoryComputing platformLaptopProjective planeState observerE-learningWeb browserMoment (mathematics)NumberBitUniverse (mathematics)AdditionPoint (geometry)Numbering schemeDirection (geometry)Integrated development environmentField (computer science)Group actionStudent's t-testCloud computingLanding pageMereologyService (economics)Similarity (geometry)Musical ensembleSign (mathematics)Electronic data processing2 (number)Repository (publishing)Software maintenanceComputing platformRight angleAreaStatisticsNormal (geometry)Arithmetic progressionBit rateConfidence intervalIdeal (ethics)FeldrechnerVector spaceWritingComputer animationLecture/ConferenceEngineering drawingDiagramMeeting/Interview
23:19
Computer-assisted translationComputer animation
Transcript: English(auto-generated)
00:00
Good morning everybody Yeah, cubes and clouds What is this going to be today? Yesterday we've learned an exciting way of how to organize people in circles in an Estonian dance that we did together So that was a really interesting Exciting and I think it was a traditional way of doing things
00:23
So today we are going to take a different approach and we're not organizing people in circles now today we're going to organize spatial data in cubes and We're going to learn the new approaches that are available. So we're trying to leave the old approaches and Do a step in the new direction
00:43
and This talk is going to be about cubes and clouds So it's an online course to educate the next generation of Earth observation researchers or generally geospatial researchers In data cubes, so that is a big topic that we've heard about in this conference Cloud platforms cloud technologies, that's also very big topic these days and also in open science
01:04
So not only how do we operate but also how do we work together and below you see? Is a joint effort effort of a huge team and I'm here to talk in their name today so
01:20
What is our aim with this online course? Maybe I should say a MOOC is a massive open online course It's really available and open to everybody Who is interested in learning new things about the topics? We just talked about and why do we need? This kind of course. So just the last talk already said
01:40
making tutorials and this stuff is still underappreciated in the Scientific community and also the newer technologies are the less chances you have to find a curated course on the topics especially since we're talking about How to organize spatial data how to operate spatial data in clouds and how to share it is three different topics that you actually need
02:06
to conduct Research in the Nowadays in the modern ways and to collaborate so having these three things tied together Is an effort that we did and I think there is no other resources currently available that are
02:22
Doing this. So this is a good starting point for everybody who wants to dive into these topics And So I said we are trying to make the transition from the traditional ways of working with in Earth observation or geospatial Easier, so I guess everybody knows you've been working on laptops probably and you are well aware of how to do this
02:45
but It's good that somebody helps you to go in the direction of how to use a cloud for some Or if you've never heard the term cloud before it can be a little bit intimidating what's happening in the cloud What are the concepts behind this and why do I have to leave my old structure behind? What do I do this for?
03:01
So I think this little barrier to jump over we're trying to get people into Jumping this first hurdle and then I think they're free again to work in whichever way they want to so that's some our idea and the target group is that our young researchers are researchers that Want to advance their knowledge in cloud technologies
03:23
So, how do we do this? How do we convey the message? There we're using in three pillars on the one hand side. We're starting to teach the concepts and This is you before you start you really have to learn
03:40
What the concepts on behind data cubes in cloud computing actually are and also behind open science So this is a theoretical approach that we speak the same language that we speak on the same definitions, which is already Difficult when you think about cloud platforms and what is a service provider? What is a data provider these things? What is a server? What is a client?
04:01
So this can be a little bit intimidating So we're just trying to get everybody on the same page there The same is true for data cubes. And why do we need this? Why can't I just work with files? What is the virtual data cube? What is the dimension of a data cube? What are labels and so on? And then the same is true for open science. And why is this important? How can I actually do open science?
04:21
What are the paths that I can take to make my data open? So what are do eyes? How do I create them and so on? So this is the first step once we speak on the same language or all on the same page we're going to the second chapter that is discovery and There we're starting to understand and interact with the technologies that we have talked about. So we're going to
04:43
Discover these clouds and the cubes that are saved in clouds via various data catalogs So where can I actually find all this data that everybody's talking about in big data? But you need an access point where to find and you also need to know how to use these tools to find the data So that's one part and also we're going to search data properties because when you're talking about big data
05:06
You don't directly want to download the data first. Maybe you need to know how can I actually filter data? What are properties that I can use for that? What are file formats that are suitable for working in the cloud and for sharing data and What are the processes that are actually available on a cloud format?
05:23
So somehow you need to know how you can actually interact with the cloud in which kind of analyzes you can do there Once We're done with this. We're actually going to apply everything we learned in a real-world scenario So you're going to start processing a whole workflow from end to end
05:42
In the cloud, you're going to validate these results and in the end you're also going to share The map that you have created in an open community project. So in this case Here you're going to make a snow map You're going to select a small area of the Alps that hasn't been mapped before apply a process by loading the data
06:03
Subsetting the data to the area that is relevant for you Aggregating the data doing some cloud masking doing some thresholding and in the end You will arrive at a binary. It's no classification But that's not done. Then we're showing how you can validate this snow map and once you've validated it, we're also going to upload it onto a public available stack browser and
06:25
The participants are learning. How do I fill stack fields and how do I publish my data in the end? This data is constantly available on a Stack browser. So when you're done, you can go to your employer maybe and show them look This is a piece of a map that I've created on this cloud that I've been using and you can see that
06:45
I can actually adhere to all of the things that I told you that I'm able to do and How do we convey all of these messages? So that was a very let's say theoretical concept that we talked about so far, but how does this manifest itself?
07:02
On the one hand side we're using lectures, of course they are on the eLearning platform EO College So we have text forms We have the three chapters that we have already talked about and every chapter has sub chapters and a quiz at the end
07:22
But that's not everything so reading Tires is tiring I guess after a while. So we've also added a series of videos I think we have like 16 or 20 videos in this whole course that are picking out let's say they're shining a light on a very special part of The lecture that we have been talking about for example the one we see here
07:43
This is the chapter about open source software and here we are highlighting The role of GDAL in the open source view geospatial world. So why is GDAL important? Why is open source software important? And what is GDAL's role in cloud computing and
08:01
We've created these videos and with experts in the field and this one was the main developer was helping us to create the video So they're very short but they're very precise on one point and just show you the application of what you're learning in the real world and Besides the videos. We also have a lot of animated content. So little games that you can play here
08:21
For example, this is some what does fair stand for and we have a couple of keywords that you can drag on to the Different letters and you will get a little score in the end. So there are a lot of keys and And That's not all so so far we are learning but then the hands-on experience is very important as well
08:40
So you're also learning to code in this course There predefined jupyter notebooks that you can execute and Yeah that guide you through different steps of programming and you have different questions after these after you've completed these and notebooks
09:00
And if you have got the right answers, for example, how many pixels are in this data cube in the end of the exercise So, how does this actually look like because there's quite a complex infrastructure behind that and trying to go
09:20
Quick through that. That's quite a technical slide, but I think it's still important to understand the effort that is behind Making such a course also Trying to make it a one-stop shop that you start at your college and you don't have to worry about Registering to a jupyter notebook registering to a cloud. So this is all explained and you go through it in one step and
09:42
Of course, you have to make choices at a certain point which technologies are you going to use you cannot Depict all different ways of programming. For example, we're using Python now would be nice to also use R But we had just to make some choices to stay within the budget But the course is open everything is available on github and we're really happy if somebody wants to contribute and we're already starting now with a
10:02
Add-on project to also incorporate other programming languages So, but how does it work is that? There is a e-learning Platform, which is the central access point. That is EO College once you register there you also register directly at jupyterhub So that is where the EO access jupyterhub. That is where the notebooks are hosted and where you have your environment your programming environment
10:29
The notebooks that are hosted on jupyterhub come from github So you can also access everything that is available in this course on github. The whole course is on github It's also on cinodo so you can download versioned
10:42
Versioned editions of the whole course and you can also reuse it. You don't have to use it in your college You can just take the markdown files render them wherever you want You can pick whatever you want. You can open issues if something is not working or you have ideas to collaborate and We also did it to show that working openly is not just taught in the course, but it's also a real practice
11:08
Then the animated content we've created on the creation hub using h5p widgets the videos are hosted on YouTube and the results that you generate on the
11:22
Cloud so we choose This time the Copernicus data space ecosystem is the cloud that you're going to log into and you're doing it with the programming language Open EO or with the cloud communication language Open EO with this API and This said we're also working on integrating the Pangeo community that you have different ways of doing things
11:46
The results you create are uploaded from jupyterhub to a public stack catalog and Are made available Disseminated by a stack browser So that's the background architecture that is completely open and can be reused also for other courses if you want to continue in this direction
12:06
so I Think it's time to go on a user journey. So now we've talked about this course a lot But now let's look together how it actually looks in fields And I hope that this is going to motivate some of you to maybe subscribe to the course or in case your university teachers or whatever
12:21
Maybe have some students take the course so the first thing we're going to do is we're going to EO College and searching the Course cubes and clouds and you're arriving on the landing page with all the informations. You have all the links here to the GitHub community to the Zenodo community to the stack browser and so on you have the description of the course and
12:43
You see how far you are in the course and how many lessons are there? How many topics how many? quizzes and So this is the main entry point Then once you start you click on a lesson and you're going to start the course. This is how the look and feel of the
13:00
e-learning platform is so you have your navigation Bar on the side now. This is yeah, I've done the course So this screenshot indicates that I'm already done with everything but actually you're unlocking every step and by completing the last one and You also see how does and how do these
13:22
Interactive animations work. So this is where you will be learning most of the time This is how you Go to the hands-on exercises so you can stay within Jupiter within your college and you just click on the exercise button and it directly forwards you to your
13:43
personal Jupiter hub environment And Where you find? the different exercises There's also a catalog that you see up here that shows all the different exercises you just click on execute and you can start programming and Here you see this is the result of the
14:02
Of the final exercise or of the validation exercise. So here you see The snow covered area that is a binary pixel map that you have created aggregated for the catchment You see you have a high Volume of snow cover in the high winter time then it's decreasing and you have low snow cover up in July
14:22
and at the same time as a validation or a plausibility analysis you can look at a Discharge That is of course growing in the river once the snow is melting then these are the quizzes how they look like. So
14:43
Once you finish the quizzes and you finished them the exercises, this is your last Exercise let's say this is a snow. This is an example from the stack browser that is online So here's the participant Carolina Guna and you see she's calculated a snow map here It's available on the cubes and clouds
15:04
Stack browser and it's publicly available and There's also this is not a the stack item, but there's also the catalog So there's slowly every participant that is doing the course the next person can maybe pick the next patch here and start classifying the snow cover in a different region and once enough people have
15:24
classified as no cover We will at some day have a full snow cover map of the Alps or of the world So that's the idea behind I'm showing what a community actually what community is and how you can share your data once you're done with this there's a Certificate that you can also put on LinkedIn and they said you have
15:43
The proof that you've done the work on the publicly available stack browser. Here's some user numbers Yeah, we have so far we have 300 users at the moment or 400 around What we have seen so far a lot of people have subscribed and stopped at lesson one
16:03
so not so many people there are a lot of people still in in progress at the moment and 50 have completed the course so far. I think that's normal at the beginning a lot of people just want to look Does this actually work? What is it? But we hope that the number of people that complete the course is going to Increase in future and as I said, this is a community
16:25
Project we also happy to see that we have participants from all continents Mostly from Europe so far that's I think logically since this is a funded project, but it's also great to see that we have from participants from other continents and talking about the target group that we've identified before so young researchers in the field of
16:46
EO or geospatial I think that is we hit it quite well there. We see that most of the people are juniors PhD candidates or students With around a little bit less than five years experience. So that's actually the ideal target group
17:03
Then we also wanted to evaluate does this course actually help Or do people feel that they have a feeling of confidence after they finish the course? So here we had some questions that we asked before the course and some after We don't have so many people that answer the questions after the course so far because it's it's not mandatory
17:21
But you see it we have a writing from one to five. How confident am I using EO platforms? how independent am I using EO platforms and How well can I adhere to open science? I Said numbers are not really matching. So a statistical analysis would be said too much But at least I think you can see a tendency that after the course, there is no
17:44
participant who is completely afraid of Using cloud platforms, especially not doing open science. So we see people can adhere to open science afterwards They feel Independent on cloud platforms and they they want to use cloud platforms in the future
18:03
So I think that is a good sign in the right direction. We're looking forward to having more participants maybe some of you and If you're not participating there are also other ways to engage in the course on your college on github on Zenodo and By the stack browser are there questions from the audience?
18:38
Hi, I'm Henry from all the University of Finland first of all brilliant work
18:43
I really like what you have been doing and actually we have been doing something similar in Finland for vector Data processing related things. So I have a couple of questions that relate to our experiences with these So first thing is that do you provide some kind of support for the students when they are doing for example the exercises?
19:05
And the other thing is the certificate So would that certificate be also applicable so that actually the students taking the course would be able to use like get credits for the university Yeah
19:21
Yeah, thanks for the questions and great to hear that. There's a similar movement also for vector processing Maybe we can combine them or at least make them available somewhere in the same region that people can Yeah work together and somehow get their learning path and through modern these technologies and The first question was do we support?
19:42
students with the exercises that's a good question, especially in the let's say in this environment cloud technologies are changing so rapidly that It's a almost a miracle that the notebooks will work for a year without maintenance So I said we use the github community or our github repository for that
20:00
We encourage also and a landing page we encourage the students to raise the issues themselves. They find them and And then of course we once we get an issue log we we discuss it with the students there So that's the kind of support that we give at the moment And I think that's also a good idea to kind of push or the people in the direction of using github if they're not already
20:25
Used to it and the second question was about the certificates So We have Calculated the hours that you usually need for the course and we have translated this into ECTS points But there's no official University, I think that will that have has accepted this course yet
20:44
universities are open to do so, but I think you can you can take the course you can Show the certificate to your supervisor to whoever and if the university says, okay, I want to accept this course They can so we have let's say we've laid it out as far as we can This is the number of ECTS
21:01
on that certificate or is it that the Universities can actually directly see because with you know, as recourses they actually write also how many ECTS you could get. Mm-hmm I think we I have to check I think it's around two ECTS and I'm not 100% sure where this is written out
21:22
I think I'd have to look to the certificate to figure it out. But yeah, I think there's a there's a diploma supplement that's the name diploma supplement we give in addition to the Certificate that shows what have you learned and how many hours have you spent on different things? And what does that translate to in ECTS? So this is something you can give to your university to it makes it maybe a little bit easier to get the credits for it
21:47
question IHE Delft Institute for water education About the LMS that you're using is that custom-made or is it also an existing LMS like moodle?
22:01
That is the e-learning platform right? Yeah. Yeah your college that is a spinner from the University of Jena and they're specializing in spatial This is a spatial e-learning platform So they have different topics especially on earth observation so far a lot of courses that have been funded by ESA end up there
22:26
So it's already a big pool How to learn earth observation from Very basic things to a hyperspectral earth observation to SAR stuff to how these translate into SDGs So there's really already a well from your college So once you go there and click through it, you will see that there is really a whole lot of
22:44
Courses already available Question The hosting of the Jupiter notebooks you use the EOX services. So is that expensive? You X was a partner in the project we also have we had Tina here at least who is a part of the company
23:05
they also have their booth over in the Exhibition area and in this course, I think they're currently still doing it in kind So that is great for this course, but for the pricing generally I don't know their their pricing scheme so far. They're doing it in kind for education
23:20
So that's great and we're happy that they're on the team