State of JAI
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 295 | |
| Author | ||
| Contributors | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/43455 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
| |
| Keywords |
00:00
State of matterUniform resource locatorWeightGeometryState of matterImage processingFunctional (mathematics)Library (computing)Virtual machineJava appletImplementationProjective planeMultiplication signWindowOperator (mathematics)TesselationRaster graphicsMassDemo (music)Medical imagingThread (computing)NeuroinformatikSlide ruleSpeech synthesisComputer simulationServer (computing)Open sourceProcess (computing)Sound effectLecture/Conference
02:08
Operations researchOracleMilitary operationOpen sourceGeometryArithmetic meanContext awarenessLibrary (computing)Java appletComputer animation
02:24
Operations researchExt functorMilitary operationOpen sourceJava appletIntegrated development environmentExtension (kinesiology)Default (computer science)Medical imagingCodeOracleExtension (kinesiology)Core dumpData miningOpen sourceMultiplication signIdeal (ethics)Right angleMusical ensembleDistribution (mathematics)Contrast (vision)MathematicsHecke operatorOperator (mathematics)Computer architectureTesselationFunctional (mathematics)Uniform resource locatorCoprocessorGeometryImage processingProblemorientierte ProgrammierspracheSpecial unitary groupProjective planeOrder (biology)Library (computing)Software developerConcurrency (computer science)INTEGRALConnectivity (graph theory)Revision controlInformationCuboidComputer fileDynamical systemServer (computing)Scripting languageGoodness of fitRun time (program lifecycle phase)Modulare ProgrammierungSemiconductor memorySource codeNeuroinformatikFinite differenceBitCalculationRaster graphicsLatent heatIntegrated development environmentDomain nameJava appletComputer animation
07:15
Image warpingPixelChainSoftware developerInterface (computing)Expert systemImage processingVirtual machineRevision controlBitJava appletMereologyPlanar graphLibrary (computing)Functional (mathematics)CodeMedical imagingMultiplication signEDIFSet (mathematics)Operator (mathematics)Mechanism designRaster graphicsClosed setGoodness of fitOpen sourceCore dumpLevel (video gaming)TesselationProjective planeConcurrency (computer science)Data structureCache (computing)Descriptive statisticsMusical ensembleReal numberComputer programNumberWeb pageEuler anglesExtension (kinesiology)Reading (process)Spectrum (functional analysis)HypercubeLogical constantEnumerated typeServer (computing)GeometryNeuroinformatikPlanningBit rateMoment (mathematics)Process (computing)Centralizer and normalizerCondition numberSound effectArtificial neural networkWeightOracleMathematicsLecture/Conference
12:37
Focus (optics)NeuroinformatikHypermediaJava appletProjective planeRaster graphicsLibrary (computing)ImplementationCodeExtension (kinesiology)Medical imagingOrder (biology)DatabaseCache (computing)Uniform resource locatorSemiconductor memoryBasis <Mathematik>TelecommunicationSoftware developerPixelSource codeTesselationConcurrency (computer science)Line (geometry)MathematicsOpen setMechanism designMultiplication signWeb pageTable (information)Operator (mathematics)Functional (mathematics)Core dumpMereologyImage processingOracleEvent horizonEnterprise architectureProcess (computing)Spring (hydrology)AdditionBitSystem callOpen sourceGoodness of fitLecture/Conference
17:59
Just-in-Time-CompilerGamma functionComputer-generated imageryLink (knot theory)Menu (computing)Java appletSolid geometryFormal languageAdditionComputer iconCodeLibrary (computing)Address spaceFunction (mathematics)Level (video gaming)Order (biology)OracleImage processingWeb pageLibrary (computing)GeometryExecution unitTable (information)Server (computing)Human migrationPoint (geometry)Direction (geometry)Computer animation
18:53
Projective planeMultiplication signConcurrency (computer science)SynchronizationTesselationLibrary (computing)Java appletCache (computing)Scaling (geometry)Different (Kate Ryan album)Process (computing)Level (video gaming)Computer clusterBenchmarkComputer animationLecture/Conference
Transcript: English(auto-generated)
00:08
Okay, so the third talk in this session will be given by Joe DeGarnet from LocationTech and Andrea Aime from GeoSolutions and they will talk about state of JAI.
00:24
Thank you. I used a 10 to 5 years ago when I was at a conference speaking on behalf of LocationTech and it just refuses to believe I work for anyone else. So I actually work for GeoCAD. I think I have a slide to that effect. Perhaps not.
00:41
And then I also work, do some volunteer work, the Open Source Geospatial Foundation and also LocationTech. So I'd like to speak with you today about a project from the dawn of time called Java Advanced Imaging and this is a really kind of fun project.
01:00
So it's a pure Java implementation. It's an image processing engine and it's surprisingly fun. So it offers all these kind of features that we use and we enjoy from GeoServer. It helps us deal with massive quantities of raster data, deferred execution, a tiling computational model, threaded computation, we can extend it, used for remote imaging,
01:26
all kinds of great image processing features, region of interest and so on. So this is a really cool library and we have enjoyed it. We continue to enjoy it and I would provide a demo but I was going to say I don't have
01:44
a Windows machine but I actually do. But you can see that this lovely demo shows some of what they were thinking of when they made this library. So this has lots of functionality that would be suitable for helping doing like Photoshop
02:00
and those kind of operations. Other things that were cool in 1999, we've got Ricky Martin, Backstreet Boys, all kinds of fun things. So I just want to put this in a social context for you for how cherished this classic Java library is. So how has JI been used for Geospatial?
02:24
We have extended and by we I mostly mean Andrea, Amy and GeoSolutions. We the community have extended Java Advanced Imaging in order to deal with our Geospatial concepts. So we've added features to existing operations for like no data, region of interest, improving
02:41
the performance because our raster files are huge and developing new operations. Kind of here is a whole bunch of the functionality that's been rewritten or new versions of these operations have been done in order to deal with our Geospatial data.
03:00
One thing that's of interest to me is we've actually taken whole central components, this concurrent tile cache, and ripped about and replaced them. This library has a very good architecture that have enabled us to do this. And there's a little JI tools project that adds like creative bolt-ons and lets you
03:22
dynamically generate stuff. Now this project's no longer active, but we have been kind of mining it for good ideas and they've been showing up in Java Advanced, J-I-E-X-T. One of the most fun examples here is called Jiffle. So Jiffle is a domain specific language, kind of lets you do like a raster calculator
03:43
and define your little script on how you want all the raster bands and so on calculated. But it actually calculates them and compiles the code right then and there into an operation that can run amazingly fast. So it's really impressive technology.
04:02
Here's a Jiffle example. You can actually put this into a little script in GeoServer and do these kind of dynamic computations on the fly. GeoTools for the developers has some integration here. How many people are like Java developers? Couple people, excellent.
04:23
And then we also have wrapped this up in a grid coverage. A grid coverage is mostly a raster file that has a bounding box. So we echo all the different operations that you do in your image processing library and we carry that bounding box information along for the ride.
04:41
Here's an example with GeoTools coverage processor doing a crop operation. So GeoTools operations, these are equivalent to that image worker, but they're working with geo-referenced imagery. And here's a fun little example. This is in a Maven file and if you go to use JI, there's this trouble.
05:03
Who can spot it? I've underlined it. So your download instructions are to go to this location and download the jar. And that's actually kind of difficult because that URL no longer exists. So Sun was bought by Oracle and a lot of these things have been kind of neglected
05:23
and lost to the depths of time, even though we use and enjoy them. Now the other trouble is we can't just pick this up and rescue it. So it's a Sun distribution license, it's free to distribute, and it's not open source.
05:40
So we can't see the source code and pick it up and change it. They do have like a research license, so you can access the source code, but for research purposes only. So this is difficult. It was originally a standard extension to Java. And the idea was they made a module system, so you had your core Java runtime and these
06:02
extensions like JI and Java 3D would plug into it. And so it looked like a perfectly good technology to use at the time. But when Oracle open sourced Java, they open sourced the core runtime, but they didn't open source these extensions, so we've been kind of left out into the cold.
06:24
So how the heck are we gonna keep GeoServer going? So we had a couple problems here. This research license was not open source, we couldn't pick it up and fix it, and the project has largely been abandoned by Oracle. And we were really quite concerned with the changes coming to the Java landscape with
06:44
Java 11. We weren't confident our library would continue to function. So we shopped around OpenCV. It's roughly comparable to JI, BSD license, but it's a little bit tricky. It can work with imagery that's larger than memory, but it's a C++ code base.
07:03
And many of our customers like working in a pure Java environment. But it's got some good stuff. Image Day, this looked much better, but sadly it's kind of abandoned. The project's really fun, it started in the 70s written in Pascal, so something even older than GeoTools.
07:22
And yeah, we also looked at Apache Commons imaging. Sadly, it never even reached version one, its last release was in 2009. But this tour of competition helped us clarify what we wanted out of an image processing library.
07:41
So we wanted a pure Java solution, we wanted it to be extendable, and we wanted to have that ability to make use of native operations if they were available. The other important one is we didn't want any artificial limitations either on number of bands for hyper spectral imagery or size of rasters.
08:05
So, what if we could take this JI extension project, where we've rewritten all the operations to work with our data, and even written core chunks of it like this concurrent tile cache. Could we just keep going? Could we just keep growing this project and
08:24
gradually replace the foundation that we started from? So it was tricky because we didn't have a set of the interfaces. They were part of this closed source API. So what we did is we started setting up a project to make a new API
08:42
that we would then port all our JI work we had done over to. So we started the raster processing engine over with the Eclipse Foundation, cuz they had lots of lawyers to protect us from Oracle. And we had our first go at it at the Bond Code Sprint. And this was really fun.
09:01
We found a computational museum and looked at the history of where all our machines came from. And then we had a look at the functionality of how it was used in GeoTools and GeoServer, and looked into the history of where our image processing came from.
09:24
So this was really looking at, if we can't bring Java advanced imaging to open source, what do we have to do to replace it? And then quickly we figured out we had no idea what we were up to. So we started digging into how Java advanced imaging actually works and what the important interfaces were.
09:41
So this leads into archeological dig. There's actually documentation. Back in 1999, developers apparently believed in documentation. It must have been before Stack Exchange. And they also had a fair bit of attitude. So the very first paragraph of this 400 page book starts with,
10:02
this document is intended for serious programmers who want to use Java advanced imaging for real projects. So it's not every document that's supposed to help you understand this library that challenges your competence as a human being.
10:21
So this was a little bit intimidating. And then also digging into this, we found like, apparently they didn't have enums or anything back in 1999. They have all these static final constants. It was like pretty soon we'd be back at C code. It was amazing. We did start to tease out the central data structures.
10:43
There's a, you can have a look at some of them here. What was interesting to us, that a lot of the tile functionality wasn't part of Java advanced imaging. It was actually right there at the Java level. So that gave us some hope that we could make our own library and smoothly integrate it with the rest of the world.
11:03
And here's a relationship between how Java advanced imagery kind of bolts into the side here of this rendered image concept. So, yeah. But we also found the documentation a bit intimidating. Like, here's an example of a description of a planar image, and
11:21
it's like a whole bunch of details. And then it says, these mechanisms are transparent to the API user. Aren't here only for your edification. Like, we got this good little Java developer. You don't have to worry your pretty head about it. And so, if only we had an expert that could explain this to us.
11:44
Do you wanna leap in and explain this to us? All right, so the idea in JI is that you set up your own image processing chain. And the image processing chain at the beginning is just a description of how
12:01
you want to process your data. But it's not like you do a read and it actually reads. You do a warp and it totally warps. You do a crop and it actually crops. It just builds a chain of operations which are definitions. The thing is, the crop at the end has an interface which is the one of an image.
12:23
So, by the time you start pulling tiles out of it and you request pixels out of it, the crop calls warp and says, hey, can you give me the pixels that I need to do the cropping? And warp says, okay, wait a moment, I need the reads to do that. So, read, can you give me the pixel that I need to warp?
12:42
And read actually reads. So, it's a pull mechanism. It's kind of interesting because you have this definition of your huge image in memory. And as the output, the encoder, pulls pixels out of it, they are read from the source, warped, cropped,
13:02
put in a tile cache and so on for performance reasons. So, you can process huge images this way without ever keeping the whole image in memory. And you could also do it multi-threaded if you wanted to. So, you have a very good basis for large image processing, eventually distributed, eventually parallelized.
13:25
So, we actually did find evidence that they did distributed computing of raster data back in 1999 and we had no idea who were these people. So, as an example here, this is just a picture of what Andrea was talking and pulling you through.
13:41
So, what would it take to untangle all of this JAI extension work we had been doing from this setup? So, we found that we had a couple implementations of concurrent tile cache to work with and we kind of shortlisted what we would need to do.
14:02
But the key thing is that this might be possible. So, our archaeological dig was starting to get into real, like, Indiana Jones territory. Could we take JAI and, like, swap in our own thing and have everything else still working? Like, would it work? This is the frightening thing.
14:22
So, you know, a couple of code examples here. I'm going to keep going. So, just when we were about to start all this, remember we were doing this with our friends at the Eclipse Foundation, Oracle had just been kind of forced by Red Hat and IBM to donate Java Enterprise Edition to the Eclipse Foundation.
14:42
So, we're like, hey, wow, there's a line of communication there. While you're at it, could you talk to Oracle and see if they would give us this code base? And, surprisingly, they said, maybe. And then, so they checked with the developers and the developers are like, sure, go for it.
15:01
And one of the Oracle directors said, yes. And then Legal said, yeah, but maybe not till next year. And so, we kind of were shocked and we waited until next year. But what was interesting to us is that Oracle was still using Java Advanced Imaging buried in their Oracle database somewhere and, like, we had no idea they were continuing to maintain it
15:25
because they hadn't made a release since 2007. Also, what year is it? 2019. What year was this library from? 1999. How long is your patents? About 20 years.
15:41
Oh, okay. So, I'd like to introduce the ImageN project. So, this is picking up the source code from 1999 and we've actually been able to talk to Oracle and get them to donate it to us. So, we had to do a little bit of wrangling. So, it moved from location tech to the Eclipse Foundation.
16:03
We settled on a license. It's going to be Apache 2 to agree with our Java Advanced Imaging extension library. And Oracle had two requests that it be an existing project and that it be that Apache license. So, we've successfully kind of restructured our project
16:21
in order to catch this code base. Here's the steps I'm trying to go through for incubation planning. So, we've got the core functionality and we had to change the package names. So, we gradually agreed on that. Our initial goal is to focus on just having a pure Java release.
16:41
All of the C++ operations depended on like a media library that was open sourced once as part of Open Solaris but I can't find the source code. So, just going to forget about that. And our goal is to have no API changes other than just changing those imports. So, a nice to have.
17:00
It would be nice if it kind of works in Java 11 without warnings and we really kind of need a logo if anyone's feeling creative. And so, here's how we've been doing. I actually have the code live in GitHub. Here's our little project over on the Eclipse page. And remember that book?
17:22
That book has been actually the majority of the work. So, we got the code to compile and work in a couple of months. I was unemployed so I suddenly had some time to work on this. I did the work in the Minnesota code sprint, OSGO code sprint. And then the library really depends on that documentation.
17:42
So, I picked up the documentation, got them to sign agreements so that we can publish it as Creative Commons and converted it from HTML to using Pandoc. And then I've been going through trying to clean up the mess. And on the flight over here, I made it about halfway through this table. I've got a few more pages to go and then I'll get a release.
18:04
So, our roadmap, I'm on 0.4. I'm just trying to wait until I get this manual in place and then I can make a tag with everything that Oracle has given us. Next, I want to make sure the library actually functions. And at that point, I'm going to down tools.
18:23
When we need to upgrade away from JAI, at the very least, we've got a foundation that we can do so. But we are going to need funding and your support in order to migrate GeoTools, in order to migrate GeoServer to a new image processing library.
18:41
But the good news is I've been trying to get this done for years and we actually have a solid step in the first direction. Thank you very much, Jordi and Andrea, and we have time for questions.
19:02
I do only take on one impossible project a year. So, this is my impossible project for this year. Do you think with the new library there will be some performance improvements possible? Oh my gosh, yes. So, remember that concurrent tile cache from JAI extension?
19:24
Somewhere on that roadmap, I'm hoping we can bring it in because it makes a huge difference. Back in 1999, they only used Synchronized. Since that time, there's a wonderful Java concurrency library which GeoSolution has been able to make use of. Andrea will tell you how much faster it is now.
19:48
So, the answer was no benchmark, but it scales up a lot better. Thank you. Any other questions?
20:01
Okay. So, thank you very much again. Thank you for attending this session and see you later. Thank you.