welcome to today's webinar on the rise of drones in Australian research space my name is Dustin Miguel de Partie from the Australian national data service and I'm your host for today my colleague Karen Visser is behind the scenes co-hosting the webinar with me this one-hour webinar is the second one in this webinar series today's webinar will showcase a success stories of drone applications in Australian research space for example in smart farming land canonic and data centers we will also provide an overview of the recent developments with the research infrastructure and see how we can leverage the benefits of the reef infrastructure as well as rural open drone data I would like to introduce our

speakers for today the presentation guru and the director and data science from the University of Queensland professor Kim brethren associate dean science faculty use of queensland dr. Ken Brown director Australian plant dynamics facility district International University now I would like to hand over to dr. guru for the first presentation

Thank You Nasima for the introduction

and giving the opportunity to talk about this topic so as nassima said Ramsey - should I go to live up return so today I'm talking about the making drone data open for scientific research so this is a basically work in progress you know what we are exploring and what are the thought process behind that so just a quick introduction when we said drones

so what is drone it is nothing but it's the aerial vehicle you know it may be controlled by the on ground remote control an on-board computer so drone in a it has got different names in a commonly used as a UAVs or the are pas I liked our pas partly because in a it's a gender-neutral name rather than unmanned and so predominantly you know if it's a rotary wings what is from Italy called as a drone and then even the big wings fixed wings in an aerial vehicles are also in this category some are pretty big which is as big as the aircraft and then which overall disc I and then gather data and the rotary wings this smaller one in it it runs in a control environment very fine scale resolution and then runs prominently in the battery and then get the data so the one of the one of the main thing about the drone is you know it become in a quite cheap to run it so that's why it's become quite popular in a various application so some of the applications

is in a drone technology was started in the surveillance and then now currently in as a favorite of the applications in the agriculture and widely in the environmental space as well I would call it as a scientific research and then it's not used in the desert disaster recovery especially in it's very difficult to send a human being so the drones are quite popularly used so in turn basically we use drone in the

remote sensing capability aspects one of the thing in this context in a we use drone basically to derive fractional cover to measure vegetative vegetation species composition to map vegetation characteristics and then to do some in stock take the pastoral stock take and then even to do the survey flora fauna survey can go survey etc so for some lot of data will be derived from the measurement that is done in the drone so if you look at you know as a favor of the vegetation is on partly because you know in the in if you take a satellite remote sensing data the vegetation composition I think most of it 2/3 of the time you know it has got the cloud cover over the earth so a lot of data is misses it Brown is a really good technology which runs in underneath the cloud below the cloud and take in a good final resolution pictures and then energy and daily such as can do the analysis so in this context you know the common sensors used in drone so in drone so basically drawn is a platform and then the sensors are attached to the to the drone so in our case you know is a fairly popular things used or the the multispectral hyperspectral lidar in the video and determine infrared sensors so from a data management perspective in a it's basically the chair it's just a favorite of a challenge partly because it's it's dynamic both space by space and time it's keep on moving at both the spatial and temporal scale and then the because of its ability to capture finer scale information in a it's it's a massive amount of data is is collected in that one and the indicator so we should have a capability to basically in gesture and then the process that they can make that data and available to the users the other part is in as a favorite of the commercial companies who work in the drone industry and even at the research aspect sort of the thing you know there is always a partnership with the commercial entity where they will run the drone and then in a collected it and give to the researchers especially there are consultants especially in the mining community in the mining area etcetera the other thing you should be aware that you know you need to you need a permit to do this from a Casa to operate this and then you should be an operator you should have a license to the operate this as one and with all this you know the identifying a data owner is important I will come to that later why it is important and if you look at our example to make this data in openly accessible you know just in a take a bit of a fare principle so these are the four aspect of off the principle in a we just see how the Drowned data will fit in set of the tank so if I take the first one in a data is adequately described searchable and should have an identifier so just to give it just to give a bit of a perspective on in the drone you know the data is basically the fight fight you know flight plans and then the you know the files of the you know flight paths and the associated field data in general especially in our case and then the raw data files from the measurement it talks and then the files and then the the log from this light and once it's the process then the all the derived products as well and we should also provide the auxiliary files like a QA QC files from the inner processing so these are the in a different files you know that is that is part of a in a data publication aspect so if you look at you know all these are the you know it's a related interrelated thing so all this data should be made available and then the especially for the provenance aspects of the thing these all these data should be made discoverable and then there should be accessible from a user perspective and then to make this enough data in a searchable and then the identifier so basically you know there should be a metric in a standard to describe this data so we so you know we use the iso standards we're not sure whether the inner for example iso:191 one five completely describe everything or maybe we mount to may have to provide ellicott customized profile that iso standard and then if you make it available as part of a catalog mannina it's a discoverable and then once we put it in a catalog we drive an identifier and then that's that looks fine and then the next aspect is the you know the one of the principal is the data retrieval using open protocol so if you look at the the instruments are the sensors that are used in phone you know it provides by default process as a raw data in a

from a different file format and an even at the publication level if you look at the in the open standard in the open data policy letter that in a all the file format should be of the open format so the file format you know generally in the you know depending on the which of the instruments you know it may be a give G ot of gaming shapefile or la s file especially in a point cloud aspect zone of the thing so with with such a kind of a veracity of P you know file format then the tools must be available as well to translate our manipulating when they analyze the data and then the inner even if the you know sometimes it may be worthwhile even to provide a program you know the our program Python program where you know they should they can access this data and then you know so that it can they can run the analysis aspects or a thing if it's too confusing so for example if somebody don't know don't have a clue about you know how to access the LS file then probably it's worthwhile to provide a program so that in a it's embed so that you know that is embedded in the embedded in the our program so that they can start writing the program so that is the one of the team surfing and the next one probably is the the the data use vocabulary and qualified reference to other metadata and then so we use a in a fair bit of a domain specific vocabulary is like a GCM B is quite popular especially keyword search once and then uh you know because the because it drone and then you have a technology as as comp so fast probably there may be you know it may be invent some of the terminology incorporated into the vocabulary so that it's accurately represented with as I was explaining the the different data types and you know when each of the data type each of the enough files should be referenced and then it should be make Taz a link and then hopefully as a process antennae file and then all those should be a queryable as well and then the the final are in a principal is basically data metadata in domain relevant community standards with clear data users license so we use a an ISO standard you know that one one five one three nine in the domain which may fit well as I said before that dinner maybe women out to create a profile to accurately represent the describe be returned and lot of the standard is fine if you want to fit at the human actionable in our discovery query and access that means at the human go and click couple of things a probably we were to do like a look at in a mission action machine access accessible in actionable discovery and exploration sort of you think we're in a submission to machine query when then that may be a issue partly because of the so much of the interrelations it fifty files and etc it is also depending on the you know the D what kind of file if we are looking at the source file then definitely that will be issue and then the talk about the data usage you know so what it says is you know you should provide as an open license thing even though you know we may say we provide the all our data in the Creative Commons by attribution the identifying the owner is a key partly because you know even at the Creative Commons Attribution the copyright subsist with the data and then you need to identify who owns the copyright a several example if you are if you are a if you are a researcher who is using a consultant to collect the data so technically if you look at that aspect the person who collect the data is the owner of that data unless you have a contractual obligation arrangement make sure that the ownership is transferable to the researcher so why this is important is that dinner for the attribution as web server sitting so that if somebody uses your data they need to know you know who is attributed as a researcher you want attribution to go to you not the person who just collected the data and and the thing is if somebody is owner they can do whatever they want with the data the IV's IP is with the owner is a what they what you want is the as a as a principal investigator view owns the IP about the data still there are a fair bit of a challenge or one of the things in a V phase is the you know it's just the amount of data set that is collected we still struggle to you know make that data the ingest the data into one place and then it process everything and then the delivery delivery of that data so we we think that the the cloud platform you know the manage platform to do everything would be a useful thing and there are a few initiative going on as well at the community level who are looking exactly the similar problems so one is the OGC domain working group and the other one is rd interests so with all the aspect you know we want we don't

want to change a complete in our data management practice of return so we let respect retrofit everything to a turn data management practice so sort of the whatever the data you know we intend to you know make the metadata and iso standards and then cataloging the geo network and then that is our stead to the you know different repositories in a discovery portals so that the data is discoverable and all the data will have a click clear in attribution statement with Creative Commons Attribution and then currently we are storing in a data in the FTP server and then make that link available and we are still in the process of you know processing a lot of data but having said that the the all the raw files are there so if somebody is really interested to use the data they can process the data as well and the other thing you know we are still working on the you know effective in a robust data delivery mechanism as well so that the user just in a comment access to data and get the data and one of the thing you know we are working on the in a still work-in-progress is the you know the D portal where a user can come and get the final products so we collect in a we have done a campaign across in eleven sites across Australia so we are working on that they called it a field data and then we are working on the portal there and all those love and size data is accessible as well especially at the light out loud so if you look at the drone you know it is quite popular in the research community you know especially in the

environmental science there is a massive uptake of the drone technology partly because of the ease of use and then the final resolution of the data it provides and moving ahead partly in a probably

worthwhile to build an IOT platform to manage the research drone data it's not at the at the institution level at the overall research level so this probably enable the interconnection of devices basically based on the you know type of sensory use and what are the application you are running and that will enable to build a common data platform so that the each of the individual doesn't in a repeat the same grappling with the same problem and if you get the if the commercial word you know this is already happening there are a lot of commercial players working in this space it may be you know officials as well and then there are even lot of the open source technology already started appearing especially management aspects of the thing so what generally what we want in this one is in a researcher has got a platform they put the sensors they collect the data and we need a platform where that data is ingested somewhere the processing happen and then the ended product derived product is basically available for the third-party researcher to to analyze that data so at the individual level you know people are working but the makes everything as a pipeline and then provide that as a service to the complete community we are still working on that one it's still a work in progress thanks for the opportunity to talk about this time Thank You guru select some excellent presentation very informative now I would like to hand over to Kim for

the second presentation hi everyone my

name is Kim Bryson thank you very much for inviting me to be participated in this I'm going to be talking about drones in research in relation to agriculture and in fact in education as well and I think my talk quite nicely segues into some of the things that guru

has been talking about because the buzz of today's agriculture is around these disruptive technologies Internet of Things big data drone technology smart agriculture they're all buzzwords and commonly tossed around and there's been a number of big conferences in the agricultural space and farm management space about how such technologies can be of value to the agriculture and to the whole food science issue of the you know producing good quality food in a sustainable way what we've done at UQ is look at how we can incorporate things

like in Internet of Things multi sensor mesh network to collect real-time biophysical data which we store in the university's cloud and we've done that around the whole thousand odd hectares of the University of Gatins regional campus which has a is a multi enterprise agricultural center and we've also incorporated a second mass information network around 10 kilometers away from the campus that will look particularly in relation to we set up a living laboratory there is what we called around how we can use this real-time biophysical data with drone technology to look at biomass of pasture in particular but that can end up going obviously to various crops and vegetation as well so this network that we set up it's very flexible we work

with technology from labellum we can look at different types of communication protocols so we're not just looking at Wi-Fi but we're looking at other types of radio interfaces that allow us to transmit data in very in real time over long distances and through buildings and trees cetera and it's a network that is basically self-healing and we have a number of these nodes set up across the campus here we go smart agriculture smart water

smart environment and smart security this last one being smart security one being essentially the ability through the network to turn things on and off or open and shut gates which of course is really important from an egg car spective the agriculture one looks at well in fact I think my next slide looks at the types of data that you can gather

from these insensitive sensors when it's installed and we have all of these nodes and most of these sensors around the place so we're collecting a lot of biophysical raw data in real time which we store in the cloud and have a dashboard to essentially access the idea is if this is done through open source so that at the end of the day people can have access to this so turn may may be interested in it other people may be interested in using it remembering it's subtropical information really for people who are interested in the agricultural space but where we're looking to certainly make that available at the moment through eduroam the people who are involved in eduroam rather than anything else we've tried it on an open network and that we found that it had a risk of the university being hacked through our system so we closed it down from worldwide accent to eduroam access so lots of data by physical data coming in and here's a diagram that tells you

that we've got data coming in from these various nodes into our into our cloud and then we're using that data within the educational environment now to develop various essentially data uses so we've got research projects in here but we've got farm operation that is something that we want to do because we'd like this to be developed far more into the agricultural farm space and of course our education component type of

data you can get you can modify all of this through our data dashboard or you can look at a QR code and get through to it so that's the start but what when we

have this large amount of biophysical data what else do we need and what for and of course we want the data for things like partial answering and management animal monitoring and management crop monitoring and management and education and all data we want to get more data so drones provides that or satellite data or remote sensing data provide that capability of putting aerial data on top of our are essentially ground and ground data collection system and if you go

back to the future we're going back to the future in this because remote sensing has been around in agriculture since the 1970s and the problem for Ag in relation to remote sensing were two main data's one was data cost of acquisition and processing the revisit frequencies the things that guru has talked about actually and also a lack of skills available in the agricultural sector for this type of type of data to be used efficiently well in a cost-effective manner and agriculture people people in agriculture can be very easily turned off technology you know what is the point of technology if it doesn't deliver me a cost-benefit in terms of what I'm doing and if it cost me a lot and it goes wrong and I can't do it when I want to do it then it's not worth it so we needed to do something about this lack of skills and what we've

done here is look at developing the specialist skills that are needed to understand spatial variability in agricultural remote sensing we've set up an agricultural remote sensing lab here at they're here at the campus and what we're doing there is we're trying to integrate students from different disciplines across the academic world so engineering students come down and work on agricultural projects in relation to things like building drones and understanding what drones can be used for building sensors and understanding what those sensors can be used for in in agricultural monitoring in particular but that does involve of course environmental monitoring so always water management site is probably one of our most recorded biophysical data sites and we we get them to do hands-on work so they understand the issues in the risk we want to use them because they're

cheap platform to carry high-resolution sensors you can see here a spatial variability in a paddock at ground level you can see a bit but when you look over here at an aerial photo of that paddock the variability is huge and that means dollars to the farmer so we want to collect that data so we can optimize production efficiency and quality and we then also want to minimize risk and environmental impact so drones from the perspective of an agricultural person could enable us to do better in the smart food production game almost certainly enables us to do better in the smart environmental management game and for us as educators smart skills development because it lots of skills involved and it's fun to do and one of the things we have in the Australian sector actually is worldwide if difficulty in getting students into the agricultural industry sector and I'm meaning across the board not just you know grains or horticulture but across the board so a grant UQ we've got five DJ phantoms that we use for teaching we've got four bespoke quadcopters we've got three bespoke hexacopters and ten mini agricultural drones which are the ones nowadays because these other ones that I've talked about the ten minions are the ones that we use and I'll just talk a little bit about them in the next

few minutes they're easy to fly and there is an article out in 2015 about why we're doing this and what we're doing the design and build principles

around these this business of getting students of all disciplines to understand what they're doing we go through a design process trial and error using open software to look at designing these things we purchase the parts for them we build them they test flies or they learn to fly learn to solder things like this undertake the project and write a report so this is a classic problem based learning or active learning scheme that we can make get students do here are the types of drones that we've investigated and used over

time you can see there's certainly date range here and as much as anything else this small mini UQ mini a drone we designed and built because we understand ur the 2 kilo limit of Casa because if we had to get every student certified we couldn't do this so this little mini egg drone which I'll talk about in a bit more details a little bit down the track is the one that we're now flying and it's with a camera that looks at getting a normal different vegetation index image is less than one kilo so the first student project was in 2013 where the

student literally took a DJI phantom with a little normal red green blue visual camera back to Fuji this is his school in 2003 this is a Google map this is his school in 2013 which is the image that you can see outline here and he literally calculated the difference in mango clearing basically so he looked at change in mango so this was the first project that we looked at then we started getting a little bit more creative we developed a drone that could

carry a multispectral camera and we started looking at prickly acacia which is a serious weed environmental weed up in central Queensland big problem from an agricultural perspective because it shuts out cattle from using the pasture underneath and we compared some of that data to satellite data and found of course that these drones creating a better resolution on the ground gave us more data it's beneficial bug drone a

hexacopter weighing around about 2 kg more when we've got this white thing underneath it designed by an agricultural science student which is now going to the has been invited to be put on display at the Science and Technology Museum in London for the next 5 years in their display on innovations in agriculture over time so this is actually an old innovation now this is about 2013 2014 but it is an instance of when people started looking at using drones so this is a box full of beneficial bugs which the farming industry the farm group that we work with wanted to see if they could develop drop beneficial bugs these bits coming out into the middle of a paddock instead of as they customarily do drive around the outside and so this was a very successful product project with industry involvement that then started really introducing this sort of technology from the education from the classroom into the actual agricultural space so that was great the net drone was another one

where we were asked to fly under tail netting for one of the largest seedling users in Australia and here is the netting you can see the lovely picture if you appreciate it this sort of color variation the grain is trying here and it was called the net really were looking to see which one of these seedlings in you know 250,000 small tube plants or germinating or not and here you could see this sort of resolution

that you could pick up from a standard R be a standard not a spectacle camera you could see the two cotyledons stage here and you could introduce students to the idea that you are looking at three different bands a little bit of physics you're looking a little bit of spatial variation because you can see where in this one where my cursor is where things haven't grown and then when you go up here at two cotyledons stage you get that interest for a student as much they can actually see a plant germinating and they can see where it has not and what you do is you get them to calculate what the cost is to both the supplier and the buyer when they get a tray of seedlings and something hasn't germinated so that links it to the economics of the producer here is our mini AG drone we've

been flying it now for a couple of years we've used it at the beginning of this year for 35 students to do some projects it's based on raspberry pi technology we've upgraded it because we found that the GPS wasn't crash op but in this sort of thing I've got 10 drones which might cost me a hundred bucks each to repair but that means that students can crash him and that that's good for a student to be able to do that and over here you can see them flying in on campus sort of

data that you can get out of this quite basic no IR or an NDB our data camera is you can pick up this information quite nicely at an operational scale so again I'm trying I'm guess I'm not at that I'm not trying to get these students to be as knowledgeable as we might think from a research perspective but an undergraduate level 3 agricultural science student they go out with this knowledge and will then be able to develop their skills in the real world after that we are developing various

sensors that can fit onto this small camera so the most recent one is a multispectral sensor again multiplex reply cameras that enable us to choose very high grade research filters of specific wavelengths when we wanting to look at something and this particular camera at the moment is being used on top of one of our I oh T multi sensor mesh network to collect aerial data of the biomass of that paddock across which of some 50 hectares I believe that has a multi sensor mesh network on it so this camera is is is now working so this is when it was being developed you can see how it's been set up for little roughly PI cameras and we use Python to make sure everything goes well this is a

drone that is no longer flying but it was an attempt at delivering a drone that could pick up the RFID tags in a cows ear from a distance and it works at about 20 meters but you know we're talking a big drone now with a big antenna and whilst it worked from an operational perspective and from the idea of trying to keep things down to a level where we're not flying above casters regulations this became essentially unusable and although I have an engineering student who wishes to try and reduce this in size this is something that we're not at the moment moving forward to but it is an idea it's an example of some of the idea to come when you start talking to the industry players and asking them what they want they they wanted something like this because normally you have to stand behind beside the cow or the cow has to go through some cattle gates and literally swipe it with a hand swipe where you've got you know a thousand had a cattle that's a really difficult really time-consuming thing to do so this is about trying to improve the efficiency of being able to look at these cows in in the paddock and the

sort of data we could get from it was was really quite good sorts of types and

research and learning involved is electronics and avionics which is not normal in an agricultural sense but is really useful if you're going to get into this side of spatial variability analysis the design and build side of things we look at programming physics maths and chemistry of course because we're looking at the electromagnetic spectrum things like spectral indices and crop growth indices and chemicals in plants so if you're trying to engage your students in learning about math which they really don't want to do you bring in something like this and it changes their perception of both learning about the math and then how it can be used in real life when it would appear to be from a classroom perspective completely dry and equation driven plant physiology animal welfare and food traceability are all issues around management that we can also talk about when we use these tools and most recently we've had these sorts

of projects going on growing sense of development for crop monitoring and diseases computer vision development for monitoring pests basically but we could be doing it in the piggery or we could be doing it out in the environment to say wildlife people we've got a master of engineering students looking at developing a robotic arm for capturing these small drones out of the sky and for autumn autonomous recharging again it's an it's an academic project probably not to not to be used operationally because of the issue of autonomous flight but very useful for the master engineering student to understand why we want that from an agricultural perspective so we want to increase obviously the reach of our drones flying and then we've had people developing automated underwater cameras to monitor phytoplankton which we can then link to our IOT and and our aerial imagery of the lakes obviously the IOT has applicability elsewhere that may not involve drones are in the equine health area and the development of this visualization dashboard for our monitoring purposes and for staff to access and as I say anyone on it your own to access some of this data that we're collecting and as I say this is a project the RFID monitoring of cattle is something that would that we're still sort of looking at but the idea is to reduce it in size as the electronics become smaller and we don't have to have such a big antenna involved which means a bigger drone and more weight etc and start starting to contravene Casto regulations so we're looking at a whole range of things both from an education and research perspective but mainly the research is about getting is about interesting students in learning about this stuff which i think is going to be key down the track in terms of capability development for our industry both agricultural environment or managed landscapes because without these kids coming in with we're going to run out of people in terms of as they get older they're not going to be able to continue to do this so that's my

presentation and I hope you enjoyed it thank you very much I can't thank you and I will move on for the next presenter Dean Brown thank you for

having me on that was a very interesting talk Kim I'm going to be talking about our work at the National Arboretum in Canberra using drones to make 3d models of the Arboretum growing I'm the director of the Austrian plant phenomics facility a new and a lot of this work was done when I was working for the AOC Center plant energy biology as a research fellow and I should first off say thanks to all the people listed there that have been essential to this project so I wasn't quite sure what

level of understanding people would have on this drones are part of as Kim point out they're part of a key emerging tool set for next gen monitoring near the environment and these makeup sensors high resolution imaging lidar and a whole bunch of other things that let us measure the environment ways we never could before and since I've been working in the phenomics field really we're interested in how you monitor the environment along with genetics so you can measure the phenotype because those are the things together that define how ecosystems develop and in a crop context you know the phenotype is essentially what we want for increasing food yields so a bit of background on UAVs I think a

bit of this has been covered so I'll go through it quite quickly but you basically have a lot of choices if you're starting a drone program from the sort of low cost exporters like these quad and exporters like the DJI phantom these things are changing so fast so I used to give this talk and say that drones were expensive and confusing and then I said that they were cheap but still a bit confusing and now they're just cheap and easy to use the DJI maverick Pro came out this year it's a very tiny drone but it still has a 12 megapixel camera on it you can throw it in a backpack and you can still - pretty decent monitoring that at the low end of the things these sorts of drones typically cost in the one to two thousand dollar range although the bigger ones like this one here cost more in the eight thousand dollar range and you get something like fifteen to twenty minutes flight time and up to ten kilogram payload for the really big ones you can also use fixed-wing drones so these give you a longer flight time something than thirty to sixty minute range but you do have to have takeoff and landing issues and they can have higher payloads but they do tip to fly faster which impacts how you can monitor with them a new system that's

being developed by people that we work with that pro UAV is this vertical takeoff and landing system and we really think this is going to be an amazing tool because it gives you both the benefits of a fixed-wing and some sort of a copter because but this with a helicopter part of it it can take off straight so it doesn't need a landing zone or runway and if someone goes wrong in them and the gas motors fail it will catch itself and land so you can put you can carry quite a heavy payload and you can put really expensive equipment on it because it's quite robust on the camera

side as Kim pointed out you have a lot of options ranging from RGB cameras the the onboard cameras on the phantom work quite well all the way up to DSLRs but you do need to to watch about what cameras are using and isolate them from vibration or you get rolling shutter issues where the camera is trying to take a photo but the sensor is not writing all the data at once and so when there's vibration you end up with bad data the sort of next step up unless you're building your one someone yourself like Ken pointed out is is the multispectral cameras that can get you in DVI and the main thing that people use now are either the micro sense acquire red edge and those are in the three to five thousand dollar range there are a lot of other sensors so you can put hyperspectral cameras on your drones or thermal and you can get you know bands in the 400 to 1300 nanometer range but they you need a big quadcopter because they're heavy they're very expensive in the data is quite hard to process so there really is a range of stuff available from really easy to use up to quite challenging depending on what what output you need the typical

outputs you get from 3d we construction software which is a lot of what I'll be talking about are ortho mosaic images so these are sort of essentially satellite layers that you can put into Google Maps or map box and then also DNS and geo Tiff's and some of the software can actually give you somewhat classified outputs or remove the ground so you only get trees can provide you with RGB multispectral or hyper spectral indices and also 3d point clouds and then you can make a 3d model of your environment as well as that's what you're interested in there are a lot of options on the 3d reconstruction software front and I'll just go through a few it's important to know that it requires a pretty beefy PC to run these things so you're looking at probably 1,500 million dollars to get a PC because you need a fair bit of RAM you need a pretty good processor and then you need the graphics card pick 40 mapper Pro which is the one that we've been using because it was about the only one available when we started working it ranges from 2,000 to about 9000 depending on what license you get and you have to pay about a thousand dollars a year for support and also as challenge that you can only run out of windows so if you want to run it on a server or on the Amazon Cloud for much bigger data processing for automating stuff you have to get the pro license the other one people use a lot is Aggie sauce photoscan I haven't used it personally but it's been well recommended the cheap version if you just want 3d models is about 60 us and about 550 academic and then pricing goes up from there and those also will run on the cloud if you buy the pro version mosaic mill is another software package it's the last quote I got from them was about 4,500 euros it comes in a lot of flavors but I haven't used it there's a free software package called visual sfm which is FSM which seems to work pretty well because it's free and somewhat open-source software you know it doesn't have all the bells and whistles the expensive ones do but if you have more time than you do money for your project it's probably a good thing to explore there are also a lot of online options and these are great for testing so if you either just do occasional flights and don't want to invest in the software in the PC or you want really fast processing or you're preparing a great proposal so you just need to get some initial data websites like these one plus you just upload your your images and give you a point cloud another data quite quickly a disclaimer this is not a complete list and all of these prices change rapidly and frequently so check check the vendors for pricing and don't take my word for it there's plenty of other information out there but you see from these sort of software you can take a fight like you see on the left side here and then you get a 3d reconstruct on the forest and I'll talk more about that in a minute so the the site where

we've been flying drones is at the National Arboretum in Canberra and this is a really great site because it's just five kilometers from Anu and has fast Wi-Fi the left side I worked at was in southern Utah and it took about five hours to drive each direction so when something broke it was you know good day and a half just to get there and back and having a field site where you have Wi-Fi access from your desk and you can drive out there in five minutes really makes it easy to test new and emerging technologies until they're ready for deploying elsewhere and also the forest was only planted about seven years ago so we have an opportunity to monitor this this forest and build a three-dimensional model of the entire Arboretum growing in into the future which is basically never been possible before so we initially it's a new research site

we installed a 20 node wireless mess sensor network Campbell weather stations for baseline data gigapixel cameras these are cameras that take hundreds or thousands of high-resolution images that you squish together to get a multibillion resolution picture we've done some lidar scans both on the ground from various sorts and then we've done near monthly UAV flights and once you sequence the trees out there because tree sequencing is getting down beyond the 10 to 20 dollar per tree range you can really measure phenotype environment and genetics in a way that gives you an amazing density of data about a space that we were never able to access before so here's a camera on the southeast side

of Australia the Arboretum is over on

the west side of Canberra is south of Black Mountain if you've been there this is our main field area this is a picture

of what it looks like from one of cameras up on the hill so this is a 500 megapixel image that we generate every hour and you can zoom all the way into that image to see the forest on the far

side there this is what that force looks

like when you fly over it with the drone and then from the drone data you can bring that into pics for ideas I'll show in a second and get a three-dimensional model of the trees so the drone

monitoring program the the goal was to test and develop a time series drum monitoring program so we could get 3d models of the trees growing we'd get time-lapse geo rectified image layers 3d point cloud and then some phenotypes you can measure like tree height area is measured by top-down pixels and color data over time and this forest for studying is it's twelve four so spotted gum and iron bark was planted about two thousand twelve and it's for Hector's so it's really a perfect size for drum monitoring because you can find the whole site in about 15 minutes it's not too hard to process and it's it's very medical to that sort of monitoring we've

really come a long way on this so in April 2013 I flew the first drone over it which was a cell phone duct tape to a drone that I'd made at home and now

we're using a really nice solid matrice pro by DJI with Derril from pro UAVs and flying it for us and really good cameras and so we're really getting a lot of solid data but just in the last four years just tech five years this technology has changed so much the typical workflow in this is fixed 4d as

a software that we use it pulls in all the images you can see here that were taken from the drone and then you can see on the left all those green lines going down our key points that is detected in all the images and there's really a lot of black magic that happens behind the scenes in these software packages and you can see on the right those are some little tiny piece of ground that the software has detected as being the same in about 30 different photos and it uses that to calculate the actual position of each photo relative to the ground those are called control points from that it creates the 3d point

cloud as you can see here and that gives

you this sort of ghostly 3d model of your entire forest you can see that there's some data missing from the bottom because the trees are the the drones are just looking down from the top and can't see around the edges of the trees the total processing time to do this is very hardware dependent so it ranges from like three to twelve hours but it

allows you to go from from pictures to point clouds to three-dimensional models of the tree as you can see here it's

important to realize how groundbreaking this technology is because you know

previously we wanted to measure the location and height of every tree it would have taken you an incredibly long time there are 2,000 trees in this forest and we can do this now in about a morning but there still are a lot challenges this girl was pointed out with working up with working with and serving up the data and so we developed we implemented a point cloud viewer called poetry on our website trait caption arc you can go there and see some of the point clouds that we have online and we wrote a software package called forced utils that runs on Python that lets us pull out the locations and height of every tree and and the point cloud data associated with that tree so this this assumes an open canopy if you have a closed canopy things are a bit harder unless you have GPS locations for your trees but now once we have the the tree locations when the canopy closes we can keep tracking them and it outputs tree height top down area location our teepee colors and a point count which is a measure of how many how many points were generated for that tree and it also spits out a CSV map that you can just see a suite file that you can just stick on Google Maps or anywhere you want that has all of your tree locations so that's

the program we ran we probably flown 30 flights over the last since since mid 2015 and I want to talk a little bit about data management because it's this is really crucial and if you're applying to anything more than just occasional slides you really need to come up with a good data management plan you want to do this before you start your surveys and it's important to consider the entire workflow right because it's not just who captures it or what happens to it but the entire process there to think of who does the surveys if you have more than one company or more than one person that data has to get to you somehow it has to get processed you have to figure out where it goes on your computer and track it you know until people like turn have made us nice tools for having our data go seamlessly online if you have to manage all this stuff yourself and it gets to the point where you publish it you have to figure out smart ways for naming things and you know you may get a data set and then add to it and then add to it again and then process it and that you need some sort of workflow that tracks how that how that is taking place and doing that across thousands of images or hundreds of flights can really be a challenge and if people upload data you need to make sure that they've told you how much data they have so that you can have all of it so you don't spend a week processing their their data and then find out that they haven't finished uploading it and you needed to add another hundred images and run the entire thing again often you run into problems like we stole all of our data on a large data server at the research school biology but we process it on a computer that's local and has an SSD so we have to move the data gets uploaded into one folder by whoever whoever took it gets moved to another folder which is the storage folder and then gets copied to this computer to process and then has to be copied back with the the new data as well and so that makes it quite hard to track things over time and you know experiments fail or new data comes in you need to have a workflow for how you know where things are and what what the status of them is so it's really best info enforce rigorous note-taking even just having people you know whatever tech is running the project writing down what they're doing is they do it can be really happy candy also shared Google Docs and notepad plus plus so you can put just files within each folder to be really useful so here's an example of

the naming structure that we settled on and basically the idea was to make when someone looks in a folder to make it reasonably human you usable so we have the year location site who captured it in the status in this case it was the National Forest in National Arboretum the ANU forest plot actually this was Pro UAV that captured that one and we wrote that it was done and uploaded so this seems like a great idea but of course if you have any sort of nested folders like you might want to name your data set National Arboretum the file names get rapidly too long for Windows and your entire process fails so this makes it a challenge because you need metadata but you have to have someone who actually is maintaining it and as I said share Google Docs are good but this isn't really a solved problem and also you know everyone always ends up with files like this because before we implemented a data management plan things were just going onto my laptop and then getting copied into random places I think I

think a lot of people get flummoxed by data management because it's really hard but it's not easy for anyone and it's easy to think that you don't know what you're doing but I'm not sure that anybody really does and you won't get things right the first time and you have to start with a plan and keep working at it and just acknowledge that it's not going to work the way you expected as soon as you as soon as you start to implement it and then you need to go back and change it because in reality our data to data management typically looks something like that and we want to move it more towards the vision but it isn't actually there but it you need to address these problems because they end up making your data unusable when we have these large scale huge time series data sets some other challenges of

processing drone data are that they're for this new these new kinds of three-dimensional data so for things like NDVI or some of the metrics that Jim was mentioning that there's a lot of known information about that because people have been working with that sort of data for a long time but a lot of data like it's a three dimensional point cloud of a tree it's hard to know how you tie data values like that to biologically meaningful things and it's also challenging getting back to what Kerr was talking about with the the provenance of the data and tracking what's been processed you know you can you can process the same project in three different versions of fixed 40 and get tree at different outputs and there's also about a hundred different ways you can vary the settings so at one point recently we decided to test every single setting we could think of at fixed 40 and you can see an output from that table here and using exactly the same images you can get a stitching time ranging from 5 minutes to 55 minutes and point-cloud sizes ranging from 44 million points down to 11 million points and it's easy to thing for example that more importance means more data but if you happen to be taking pictures on a windy day and your tree is moving around you probably want your data doesn't have the resolution of a leaf it just has the resolution of the height and structure of a tree so it may be that either the less point is actually a better measurement of that tree volume or somewhere in between but it's really hard to graduate these things because there isn't any way to go out and measure that it was a tree anyway else it's also

important to choose the right tool for the job so drones are really great people are using them for good reason with lots of things but they're best suited for smaller areas like maybe less than 20 hectares and a good example of this is that because we've been flying the Arboretum so much we'd all wanted to do a full Arboretum survey so that we could get a 3d point cloud and model the entire Arboretum and measure the height of every tree in the Arboretum I think it's about 35,000 trees that they have out there so we finally got the funding to do this we got jerilyn to fly it and it ended up being a huge project it took many weeks of planning there were four or five flight days required you have to have multiple staff on-site because when you find an area that large you can't fly over people so it's easy to follow council regulations when you're just a remote forest but when you're trying to find fault fly over a 250 hectare area that's open to the public it becomes much harder we ended up with 8800 RGB images and more than 12,000 images from the Sequoia it ultimately took about two months of manual processing because we had to break everything into smaller subsets because the full cloud couldn't run on any machines none of the online folks can handle more than about 500 images so we couldn't just throw it at one of the cloud services and we ended up with a point cloud that had 584 million data points so it is important to consider what work for you will use so this is something we know we can probably do this once a year max but if you add up the time cost of it it becomes prohibitively expensive to do this at this point with the technology available and for something like that it might be better to fly a plane over the Arboretum for instance it's also if you're thinking of setting up a monitoring plan you need to consider weather and distance the decided accessibility and time of day because if you want to fly your all of your plots at the same time of day say around noon you have to drive between sites you can't do that on the same day and sometimes it turns out that just putting a camera in so you can get consistent data even though it's not as maybe high resolution might be a better option or using satellite data and again for example a lot of the local agricultural monitoring it turns out it's cheaper to do it by helicopter because you might want to fly five flight fly five sites in the same day and the helicopter you can do that in about an hour it might cost you a couple thousand dollars but that's cheaper than trying to drive around to five sites over the course of a week and flying a drone and you can put a much heavier payload on a helicopter so this is the

point this is the point cloud we got out of the Arboretum again if you go to trade capture you can you can go around and explore it it looks pretty good but you can see there are some artifacts where the stitches didn't line up perfectly because we're doing had to do it in pieces so again on choosing the

right tool for the job larger scale surveys are not necessarily best to do with a drone and there was interesting article from frontiers in the ecology the environment last year where they looked at doing a UAV surveys in Alaska and it turned out that it was 1700 per site to use a UAS but only four hundred dollars for site to use a plane mainly because the sites were quite far apart so it's important consideration now in the future I think

what we all want is self-driving drone forms that fly over our forests or field daily we have all this data upload and process in the cloud and get near real-time analytics to the farmer or the end-user and I've been wanting someone to release something like this for years and finally this year a country called X aircraft in China and a group in Sydney cup revolution AG are starting to release the system so here's an agriculture UAV I can carry pesticides or water or nutrients fertilizer and they can fly themselves so of course their regulatory issues around this but we're getting close to the point where we can have a swarm of drones that's just monitoring our sites continuously and in the future as micro drones get

get better and higher-quality they may be more feasible and safer particularly for field monitoring where maybe they aren't getting people there because you can imagine a small small study you pay ten thousand dollars and get five hundred at these drones and they take off every day and fly around your forest and come back they're all solar-powered they live on a tower somewhere that would be a really amazing processing solution to give you a continuous three-dimensional data of your of your research site so some thoughts on

developing a drone program drones and sensors are crucial component for field monitoring you're see no typing but it is hard to do there's a lot of technology available so you want to focus on deliverables who are your stakeholders or customers and what do they need to know what would be actionable information for them it's easy to say oh yeah let's just going to drone and then realize afterwards that you're not even really figured out what kind of data you're going to deliver deliver and then working back from that you can figure out how big the area is what the full costing is and so on it covered a lot of this and then also you want to start small and get your pipeline working so you don't want to start off line the 250 hectares or the Arboretum you want to have done the smaller force first you want to make sure that UAV you buy is the correct one and whether or not it might just be cheaper to subcontract and when you process the data you need to decide if you're going to outsource it or do it yourself and if you need to develop novel tools are there groups already doing this stuff and how can you share this with others so that we all have good tools user processes and sorts of data and again don't underestimate how hard data management is and have a plan in place at the start I think we're

running out of time so I'll run through this really quickly too

I just want to say that all of this data is incredibly its new and it's incredibly hard to manage because it's three dimensions and it's got multiple layers and time series and we don't have tools for this that that we used to like the tools that we normally use for data visualization management are not usable for the tool sets that we have now so we need a sort of MATLAB Excel or gif for time series three-dimensional hyperspectral data and this doesn't exist yet so there's a couple groups

that NCI that are developing point cloud

viewers there's a we're working with the viz lab to make a time-lapse virtual reality and Windows based point cloud viewer and if you if you search for atom skier at NCI he's working on a NCI back one which would be really great because then groups like turning us to dump all of our data in the same spot on NCI and we'd have these real-time tools for pulling point clouds out on the fly and just the last thing I'll go through this

quick because I know we're out of time one thing that I want to do once I start getting through dimensional data is be able to visualize all the sensor and point cloud data on the landscape where it was collected because that's a way to me that helps helps you really see the the data in the context of where it was collected and and the place that you're monitoring so I collaborated with some groups with some students in the computer science department and we made a virtual reality three dimensional model the National Arboretum using all the drone data as well as our sensor data essentially what you can do is you

can take three-dimensional model and software developed by Hollywood and use it to reproduce your landscape but rather than using the whatever data they're using for a movie you can use your data from your drone flights to generate it three-dimensional models and so this is an example of the the project

for the National Arboretum here's a

drone the drone flight data is the three-dimensional force that we get out

of it this is the virtual reality

version of it where we've taken the digital elevation model to have a generate the landscape and then put the

trees in at the locations that the drone data measured when you interact with the

trees they show you their metadata so in

this case for showing height an area of each tree and then you can also map onto

the landscape and play back in time

series the the different data types that we're collecting with the mesh sensors so this is a really great tool for pulling all of these dense data layers together and visualize them in one spot to help us make sense of this incredibly complex data and I think we should all

consider this is just the beginning

right so when I was a kid I used to play Atari and this is what it looked like and now this is you know the same dragon

games that my kids play and where the Atari stage in VR and in our ability to measure the world continuously in 3d and in 10 years VR and AR can indistinguishable from reality so the question is what do we do with these tools and how do we create the next generation interfaces that facilitate a consistent research and how we build monitoring and monitoring programs that make best use of all these data types so we can really model our environment in three dimensions and solve the Grand Challenges of this century there are

lots of people to thank but I think amount of time so thanks excellent presentation time to thank you

for joining us and look forward to see you again in the future