Czech Drought Monitoring system – a journey from manual work to global drought monitoring and machine learning, powered by Python
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 112 | |
| Author | ||
| License | CC Attribution - NonCommercial - ShareAlike 4.0 International: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this | |
| Identifiers | 10.5446/60787 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
EuroPython 202222 / 112
5
10
14
17
19
23
25
29
31
32
34
44
47
51
53
54
57
61
69
70
82
83
88
93
94
97
101
105
106
00:00
Software developerCodeInheritance (object-oriented programming)DatabaseUniverse (mathematics)Graphic designDifferent (Kate Ryan album)Software developerNeuroinformatikDatabaseObservational studyFlow separationBitTrailExecution unitData miningMereologyInformationMultiplication signString (computer science)AreaPoint (geometry)State of matterProcess (computing)Group actionCASE <Informatik>Presentation of a groupMappingDecision theoryPhysicalismType theoryPhysical systemLine (geometry)CodeDialectSoftwareComputer programmingCodeInheritance (object-oriented programming)Right angleComputer animation
06:35
HypothesisMereologyCodeNP-hardCAN busProcess (computing)HypothesisDirection (geometry)RoutingMultiplication signMereologyBitCodeWeb 2.0Cartesian coordinate systemDigital photographyTexture mappingPlanningLevel (video gaming)Software developerFocus (optics)Real numberFigurate numberMathematicsView (database)Goodness of fitPoint (geometry)Interactive televisionWeb portalPower (physics)Uniformer RaumJoystickObservational studyDifferent (Kate Ryan album)Scripting languageArmThermodynamisches SystemControl flowRow (database)DistanceECosTerm (mathematics)Core dumpFormal languageCodeRepetitionComputer animation
14:31
PredictionThermodynamisches SystemMoistureData modelSineCodeWage labourElectronic mailing listStructural loadPrice indexTask (computing)FreewareWebsiteHecke operatorLatent heatLink (knot theory)GeostatisticsModemMappingAerodynamicsTabu searchProcess (computing)CalculationCodeWage labourTraffic reportingMultiplication signSoftware developerData storage deviceWebsiteProjective planeFreewareProcess (computing)Computer configurationCodeProduct (business)MappingPhysical systemDirectory serviceLaptopPolar coordinate systemEndliche ModelltheoriePoint (geometry)SatelliteWater vaporScripting languageRandomizationRevision controlOcean currentState of matterMetreState observerTranslation (relic)Different (Kate Ryan album)MereologyEstimatorGroup actionPredictabilityTexture mappingTask (computing)Subject indexingVideo gameCalculationCivil engineeringSoftwareFocus (optics)BitMeasurementFlow separationInternet service providerCharge carrierCausalityElectronic mailing listDigital photographyData managementDigital electronicsOperator (mathematics)Computer animation
22:08
Process modelingSource codeThermische ZustandsgleichungSoftwareMessage passingSoftware developerBitCodeWebsiteView (database)Multiplication signDivisorAlgebraic closureEndliche ModelltheorieControl flowDisk read-and-write headComputer animation
23:38
Software developerEndliche ModelltheorieForcing (mathematics)BitScaling (geometry)Self-organizationCycle (graph theory)Process (computing)Social classDirection (geometry)Multiplication signPower (physics)Computer animation
25:11
YouTubeStreaming mediaMoment (mathematics)Direction (geometry)Roundness (object)File viewerTask (computing)View (database)Video gameSheaf (mathematics)Endliche ModelltheorieLecture/Conference
26:17
XML
Transcript: English(auto-generated)
00:06
So, my name is Monika, once again, very happy to be here, and today I would like to talk not just about the Czech drug monitoring system, even though that's something we do in our work, and that's something that's powered by Python, I would like to say,
00:21
but also and mostly about my journey, how I even came to the coding in Python, and how it was for me, and hopefully it would motivate those of you who might be the beginners or maybe a little bit lost in the tech world to navigate through all of that and find possibly your path to not necessarily becoming a developer, but becoming something else which you can
00:45
use Python or coding for. So let's dive right in. What is this talk going to be about? So as I mentioned, you don't have to become a developer to code for a living and actually have fun. This is what my case was, and you don't have to be scared if you don't really enjoy all
01:05
the technical details, you still can do amazing stuff, because you can sneak up to some other industry or research and really connect the coding with something you love or you find yourself passionate about, and you don't really have to be a tech guru to do this
01:22
and to be really helpful to some scientific team or industry team that just needs somebody to help with the coding, and maybe you have some good practices to bring from the developing world and you can still do pretty amazing stuff which I am about to show you as we do in the Czech Republic, and you don't ever have to be scared to show off what
01:43
you have learned so far, even if you don't feel like it so much, because it was also my case. So unfortunately there is some things that are not in this talk, so this talk is definitely not about some super advanced Python state-of-the-art coding. I'm very sorry for that, if there are some tech gurus in the area, please run away,
02:05
because you would probably be terrified with what you are about to see. We are also not about newest technologies. To terrify you even more, some of our code still runs in Python 2.7, nobody is running away, okay, great to see that.
02:21
And it's not about Kubernetes, whatever it is that, but please somebody tell me what is that, because I really need to know, everybody are talking about it, I have no idea, I'm just laughing and nodding, and please find me at lunch because I need some information about this stuff. Thank you. Now, to the talk.
02:40
Nobody has left still, so a little bit about me to start the journey. As mentioned, my name is Monika, I feel myself to be a mediocre Pythonista and I decided to use this title a few years ago when I realised I don't have to Google how to slice a string. I just did it, and it was right. I just got the part of the string that I needed for the first try.
03:03
So I thought, you know, I'm not a unor anymore, I'm a mediocre kind of Pythonista now. I'm also not very smart. I mean, I got into Dublin without any troubles, and not the major ones, but you know, I'm not this kind of a very brilliant brain who is still filled with thoughts, overthrowing
03:25
with ideas. I'm just sitting there watching, learning something new now and then, but not so very brilliant. Keep this in mind during the presentation, please. I'm also, as already mentioned, want to be a PhD because I decided I would do my PhD
03:42
in climatology in my 30s for several years of saying that I would never do my PhD, so that also says something about me. I am a GIS analyst, GIS stands for geographic information systems, if you are not familiar with that, and I'm also a very uni-researcher.
04:02
To sound at least a little bit fun, I'm also a winter sport enthusiast, so you can usually find me somewhere cold, or at least somewhere far from the sea level. Now if you, when you get to know me a little bit better, I can start with my journey to coding. Also known as how I learned to code even if I didn't want to, I also tried to illustrate
04:25
this path with a piece of art of mine, the crooked line that I did myself yesterday in the evening, so just saying, not only the coder and the researcher, but also maybe a graphic designer, we will see, we will see. So my path to coding started actually at the university.
04:43
I did my studies at Maastricht University in Brno, which is the literate thing in the Czech Republic, it's not Prague, it's Brno, please, just mentioning it, because it's a big difference, and I did both of my bachelors and masters in cartography and geoinformatics, and don't make me wrong, I loved my studies, it was amazing, I learned
05:04
a lot, there was an amazing group of people, it was like seven of us in the program, so it was pretty small and cozy, but there is a little bit of confusion with the informatics part of the name of the program. So most of the time, we felt like this is a monk on the image, we learned a lot,
05:25
we learned a lot about the physical geography, that means how the atmosphere works, what types of soils we have and why is that, how did it come together. We also learned a lot about the history, about old maps, how and why were they made, and who made them, how the geographical names went to the world, and stuff like that.
05:46
We also did a huge amount of regional geography, and that's why I know that Pidurutavagawa is the highest peak of Sri Lanka, and I will never forget it, for some reason, but we also did some GIS, that means clicking in the software that specializes in somehow
06:02
analyzing the geospatial data, and they also enlightened us by the fact that the databases exist. Some databases, I don't know what it is, they are somewhere, you can have data in them. And that's about the informatics that we got into. So really, we felt quite a bit like the monk on the picture, with this quill, trying
06:25
to map the world, even everything is already mapped, being coached by some other guy who is pointing somewhere totally different. More or less we felt like this is not really what the job market expects from us after we graduate in geoinformatics, because from what we got, this was maybe the picture
06:44
that job market had about the geoinformatics, and we didn't quite look like that at all, so we got nervous, and that's why, when the bachelor's thesis came, I decided I will go a different direction. For some reason, they let us do non-research thesis, and they let us do applied thesis,
07:06
so you could actually make something, you didn't have to discover stuff. And what they also let us do, for some reason, was choose our own topic, so guess what I did? I went for that. I did an applied thesis with my own topic, and for some reason I was surrounded by photographers
07:24
at this time. All my friends were doing something about photography, so they gave me this idea to do the web map application that photographers could use, to share the spots to take pictures in, everything. There was an interactive map, there was logging, there was like doing stuff with
07:45
the map actually, making it interactive, putting points of interest into the map. I remember that monk from the picture, in the last slide, oh, he suffered a lot with this little quill, so it somehow went through. For some reason, it was 2013, so I chose PHP because it was how we wrote in that time,
08:07
and I felt like it was coming together somehow. I was not happy about it, I quite a lot hated the idea of coding, and there was some crying and stuff like that, but I somehow put it together, and you won't know what came
08:22
next. Yes, this bad guy. I was suffering a lot, a lot, and I hated it even more when I have to do this part in JavaScript, which was basically everything from the map interactiveness to logging, you know that you are developers mostly, so it was terrible, but somehow I made it
08:42
through, and I gave the thesis in, and I also got into my masters because on the other hand I really loved the studies, I loved to learn, so I went to my master thesis, and you know what, my advisor somehow remembered what I did for my bachelor's thesis and
09:00
suggested that it would be a great idea to help another guy with another web map application, and I went for it again. This time it was something that already existed, but what was needed for me from the old little monk was to do the advanced geospatial stuff and figure it out. There was this huge portal for hikers to plan their routes, plan their trips, so I
09:24
had to think about how to even do a route planner, how to do the elevation gain maps, and stuff like that, so nothing too great. I was totally in the dark, thanks to my friends and a lot of good advice I somehow managed
09:41
through. The other thing was that the portal already existed and it was written in something, and it was Ruby, and I was suffering again for some reason, and at this point I wasn't even sure that coding is something that I would like to do, because I had this PHP behind me, and I didn't like that. Then the JavaScript part, I hated it totally, and then this guy also didn't speak to me
10:08
that much. But again, somehow, things got sorted out, I went through, and since I was at my masters, I got the time to get the part-time job which I got. I got the part-time job as an IT specialist.
10:22
That monk was an IT specialist now. I actually applied for a GIS job in the company that figured out how to do flood management plans for little cities, municipalities, and I applied for a clicking in GIS and doing those plans, actually putting them together and communicating with the majors.
10:45
But my upper level heard about my great success with those map portals and coding and stuff, and offered me to join the IT team, and I said yes, because I don't know, I mean, now I'm happy about that, but at this time, I don't know what I was thinking.
11:01
But I went for it. I finally learned a little bit of Python, because the company was based on Python coding, and things started to make sense a little bit. I discovered Python. I liked it much more than what I have seen before, for some reason, I don't know, maybe it was just a better collective, whatever.
11:22
I had more time to do that. It doesn't really matter at the end. Things started to turn out a little bit better, because I realised that Python and GIS technologies are actually very good friends, and that I could do a lot of stuff that I did manually in GIS in Python.
11:41
The question is why didn't they tell us at uni, why they didn't have a Python curse there? We don't know until this day, but whatever, we found our ways, all of us are doing just fine, nobody is a monk, so yes, also, a very great thing to me, thanks to the fact that I was in the IT department, I landed in the coding social bubble, so now I was surrounded
12:06
not by the photographers, which was also great, but not so helpful, but by people who do code, and they actually enjoyed it at some level. So I joined PyOADs, at first as a trainee, and as the time went, I also became a coach,
12:22
but that's another story. So, you know, I started to get around a little bit better, I finally wrote some decent code, and I also gained some view on how it should look like. What are the good practices? I'm not saying that it was something amazing, but, you know, it started to look like a code, not like a mess in some kind of language, but I also realised I still don't like the
12:46
hard-core technical stuff, the things that guys were talking about by the coffee machine, all those little bits and pieces, I wasn't interested in that still, so I thought it might be a problem, but I figured out it might also be or not, because by the time
13:02
I actually went for my first real job interview, it was just when I ended the uni, and I got into the great place, the scientific world, which I also didn't know by the time, but now I see it, and I became a part of the scientific team, because lucky for me,
13:21
I was successful at the interview, again, don't know why, but I was, and I finally realised what can I actually code and enjoy it. So, I landed at Czech Globe, that's the long name of the institution that you have already heard, a little bit about that, because it's my home for six years now, and I still
13:40
really dig it, it's the research institution under the Czech Academy of Science, we are quite big, there is 350 people, most of the people are scientists, the focus of the institution, oh, okay, is the global change impact on ecosystems, it's pretty wide, we have lots of teams,
14:02
and my team is focusing on the climate change impacts on agro-ecosystems, that means we are focused on agriculture, how does the climate change impact the yields, the pest outbreaks, et cetera, et cetera, on the picture you can see one of our experiments, when we are torturing the plants with different CO2 levels, which each one of these
14:23
greenhouses has a little bit different CO2 level, and we are searching what is happening to the plants that are living inside of the greenhouse, so most of our work is just a torture, some plant, basically, but yeah, that's what we do for science. So, the flagship of what we do in our team is the Czech Globe monitoring system,
14:42
you can find it online at the website that is right here, the English translation is not very good, we are currently developing a whole new website, but it will take a year maybe, so bear with us, it kick-started in 2014, in 2015 we had a pretty severe drought in the Czech
15:00
Republic, and we needed some tool to monitor it, because the impacts were huge, the yields were dropping, there was no wheat at all, et cetera, I joined in the half of 2016, so quite at the beginning of the system, we are monitoring soil drought, that means we are all interested in how much water is in the soil to be used by the plants. The portal was made for the farmers,
15:25
basically, because it's a group that's the first who is hit by the drought, and also for the general public, we monitor in Czechia, Slovak Republic, and also Central Europe, and it's a live product, we have daily fresh data. We have three main data or parts that we build the system upon, one is our soil moisture model,
15:47
which takes the ground measurements, and at the end of the model, it gives out the estimation of soil moisture in the 500 by 500 meters pixel, we also use satellite data, of course, of observation from many providers, and last but not least, we have our farmers,
16:05
who are reporting us the current state of drought and the impacts on the crops. So, as you can see, it's quite a data-heavy system, we have all of this model data, earth observation data, which tend to be quite hefty, we have the reports, the pictures,
16:22
the predictions, it's a lot, and I realized that besides me, and I just joined this, there is the one GIS analyst who was clicking everything in the GIS software, so to be honest, I thought I was a great fit, I had some experience with automation and coding
16:40
from my previous career, or how do you want to call it, and I also felt like I can really help even if I don't know a lot, because the other GIS analyst had no time to do stuff and automate, because she was buried in the research and the papers, because you have to realize that the
17:00
system is developed by scientists, which is pretty much something a little bit different than the commercial developing software, you don't have the product itself in your focus, you are focused on the research, and you get to a system like this just when the research papers are published and grants are, you know, granted, and it's basically a free time project for you,
17:23
you have no software development habits, you have no version control, the scripts are running at the random notebooks and PCs somewhere at another city on the GIS lab, and you don't care how the code looks like, which was new for me, because, you know, I just was on the top of
17:42
making things pretty and functional at this time, but this is what I realized, and I also realized it's my time to shine and also learn heavily, because I could actually start a code to make something better. And I decided I will focus on getting rid of all the manual labor that
18:03
we had, that I saw there, and that's because I had what I called a newcomer luxury time, you know, nobody takes care about you, you are new, they just point you to the coffee machine and nobody expects anything from you for the, I don't know, first three or four months, you can just sit there and look if you are not very into it, and I decided I will use the time
18:24
to watch and learn, but also to find out the things that I really don't want to do when I had my place in the team, and get rid of those. And I identified three of them. First was daily data downloads. Somebody was downloading data daily by hand, going on
18:43
some website and getting the data and putting them in some directories. That was terrifying. Not as terrifying as daily production of map layouts when you did the map in the GI software, also by hand, by clicking and putting things together, also terrible, but the most terrifying
19:01
is that somebody processed data weekly in Excel by hand, by copying things and doing the basic calculations, so at this point I was pretty terrified too, because I don't want to do these things when I get into the team, because these were the first things that I would have to do. People would like to get rid of it, right? So, about the daily downloads. My task was
19:24
to add the daily soy water index data to our website. That's not really that important, what the data was, but I figured out that if I automatically download these data, I can do it with whatever data we are actually using and get rid of those manual downloads. Just FYI,
19:43
if you would like to get your hands on some spatial data, there is a Copernicus data store which is ESA-covered project and you can get free EU and global satellite data at this website, so just note for you if it's something that interests you. So, I went through the daily
20:01
downloads. How do I download the data? There was a lot of questions. Don't make me wrong, even if I was motivated, I still didn't know a lot. How even get into the data formats? How do I manage projections in Python? Basic geostatistics? Nothing learned at the uni yet. How to get maps from the data? Also had no idea about that. How do you get maps to our website?
20:25
Also no idea. But you know what I found? This one. All the answers at the one website, and you know what I found else? Somebody has already done this. I'm not the first who is approaching these problems. Surprise, surprise. So that was great. I took this one down. Another
20:44
one was the production of map layouts. Weekly, daily, map, prediction layouts, terrible to do by hand because you have to produce all those little maps and then put it into one map and then put it somewhere. So, again, I went for it, had my questions, a lot of my questions.
21:02
Again, I found out that somebody already did that, and in my career, I didn't find anything that somebody didn't already do, so that's a great thing, and I realised I love it. The job was just opening the doors for me, and there was a lot of options which way I would
21:20
like to go. It was not boring, and I enjoyed coding for the first time, and I didn't know what was about to come because there was the Excel sheet data. Just to give you a picture, about 1,500 pharmacy reports weekly processed by hand, copying from one Excel to another,
21:43
semi-manual calculations, one weekday dedicated to this, and one colleague on the edge of losing her mind because she was doing it for two years now. Terrible. I decided I don't want to do this when I'm done, so what do we do in the Python community when we see
22:03
this bad boy? Yes! We go pandas, and I have to say that this specific task was something that I got caught on. I fell in love with data science, not when I was doing this, but, as the time went, and I found my calling in some way, and that's just because I went away
22:26
from the developing world and went somewhere else, so maybe that's my message here. So, what are we up to now? We have implemented some software development tools, actually. I went through and learned about Git, or teach the guys at our team about Git,
22:43
so now we have that. We also have some code reviews if we want them, and, mainly, we are still focused on science. We have lots of Python-powered science. We are predicting eos using artificial networks. We are getting together with firefighters to find
23:03
out what happens with the wildfires if they are happening near the drinking water reservoir, which is a super important thing to have a look at. We are also predicting the pests outbreak, to give the heads-up to farmers, and a lot more. You can find everything on our website
23:21
if you're interested. So we went back to science, but now the science is powered by Python. I would like to take a little bit of credit for that, because, as I learned a lot, I also had the opportunity to teach others, which is great. So, this one is something that
23:41
I'm particularly proud about, because we went global, and you can find our global model of drought at this website, which is window.com, which is an amazing tech company from the Czech Republic. The portal is just amazing. Check it out. We joined forces, and I think it's really great. To wrap it up, what to take away, if you're a beginner, do not, under any
24:02
circumstances, underestimate your skills. If you're feeling like you don't know a lot, you know, and you can take it and make something better with at least a little bit of skill. If you are scared of the developing world, don't worry, it will go away, or you can go to another world, in another direction, and you don't have to do all those
24:24
hard-core tech stuff, as I did, and don't stop exploring. If you don't like one technology, go to another, or you can do, you know, something completely different. It's up to you, because you know much more than you think. That's about it. I would like to thank you
24:40
for your attention, thank the organisers for a great job that they are doing, and having us here in Dublin, and also one personal note, I would like to thank all my friends who, in the years, tried to explain to me how the cycle within the cycle works, because I was stuck on this for half a year, and, thanks to all those explanations, I finally got through.
25:05
Thank you very much. I'm here open for questions. Thank you very much for your talk, and for the experience. It was very inspiring at least for those people who are working at companies that could do the same approach to the manual task that they're doing. Are there any questions in the room? We still
25:28
have the opportunity for the remote viewers of this to ask questions as well, but at the moment I don't see any of the remote questions coming in. What we did is we also have
25:42
a special room in the Liffey section, that's the Liffey boardroom, to where Monica will be now for a few more minutes, so if anybody from the audience here would like to ask a direct question, or if anybody watching the YouTube stream would want to ask a question directly for the next minutes, then that would be the place where you would find
26:04
Monica. So, thanks again for sharing all your experiences in introducing Python into your life, and let's have another round of applause for Monica.