Debian Java: Insights and challenges
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Subtitle |
| |
| Title of Series | ||
| Number of Parts | 561 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/44222 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
00:00
Java appletSoftwareBinary fileOpen sourceLine (geometry)RankingInstallation artPlane (geometry)BuildingIntrusion detection systemServer (computing)Bit rateRevision controlAlgebraic closureDisintegrationPhysical systemAreaFunction (mathematics)Software developerInternetworkingData modelInformation securityTask (computing)Modulare ProgrammierungIntelligent NetworkSoftware developerSource codeData compressionFile archiverINTEGRALMereologyMultiplication signCuboidScripting languageGame controllerSoftware repositoryCorrespondence (mathematics)Line (geometry)Open sourceLatent heatAlgebraic closureCountingJava appletPhysical systemBoilerplate (text)Point (geometry)Formal languageVirtual machineOnline helpOperating systemLevel (video gaming)NumberGroup actionE-bookBinary codeLibrary (computing)FreewareProjective planeRevision controlBuildingConfidence intervalAndroid (robot)Process (computing)Cartesian coordinate systemSoftwarePlanningInformation securityComputer fileServer (computing)ScalabilityPosition operatorScaling (geometry)InternetworkingCASE <Informatik>Software maintenanceTraffic reportingSoftware bugComputer animation
07:42
Open sourceRevision controlInformation securityCodeSoftware maintenanceJava appletDiscrete element methodServer (computing)Information securityVideo gameProjective planeStandard deviationPatch (Unix)MassRevision controlBitSoftware developerGroup actionOpen setGraph (mathematics)CuboidSoftware bugCASE <Informatik>Run time (program lifecycle phase)SoftwareMultiplication signLibrary (computing)Physical systemStaff (military)Axiom of choiceVulnerability (computing)Cartesian coordinate systemMiniDiscTask (computing)Different (Kate Ryan album)Validity (statistics)Right angleSpacetimeFiber bundleOpen sourceSocial classSoftware maintenanceDistribution (mathematics)BuildingData compressionXML
15:17
Point cloudComputer animation
Transcript: English(auto-generated)
00:06
Hi and welcome, my name is Marcus Koshany and I'm a Debian developer and also a Java team member. Today I want to give you some insights into the Java in Debian
00:21
and what challenges we face during our daily packaging work. Java is a very important language in Debian. If you just count source code lines, it ranks in third place only after C and C++ which are used in two-thirds of our software packages.
00:45
With more than 19 million source lines of code, there's slightly more Java code in Debian than Python. I know what some Pythonians would say, it's because of the boilerplate and they are probably right. But still, it is a very important language
01:03
and it has a significant number of source code lines, three times more than Perl. Perl is a very significant language in Debian because almost all our most important system tools are written in it. The Java team alone maintains about 1000 source packages
01:22
or another count over 1600 binary packages. This is almost 11% more than we had in Debian Stretch. Of course, we are not the only team which deals with Java. There are other teams like Debian Met or Debian Science
01:43
who deal with specific software packages or software written in Java. For example, for scientific research, bioinformatics or medical care.
02:01
Here, we have also a dedicated team for Clojure and there's another team for packaging Android software. So if you ever wondered how you can build the Android operating system from source, there's a dedicated Debian team who does only that. According to our popularity contests which is opt-in,
02:21
more than 40% of all systems report that they have an OpenJDK 8 installed on them. So you can see there's a lot of demand for Java. Popular applications that use Java are LibreOffice and Parts,
02:42
NetBeans, SpeedForm 3D, Freeplane or Freecore. We expect similar numbers in our upcoming Debian 10 Buster release. What can you expect in Debian 10 Buster? So first of all, very important,
03:00
we have completed the OpenJDK 11 transition which required more than 400 package updates. Actually, I wanted to complain a lot about the grief that was caused by the transitions but someone told me to keep the talk positive and I shall not rant so I skip that part for now.
03:22
Let's move on to build tools. Ant and Maven are up to date. We have some problems with Gradle. It is stuck at the last pre-Kotlin version because the Gradle developers decided to rewrite some of their build scripts from Java to Kotlin. It sounds like a simple problem
03:40
but for us it means we have to package Kotlin to build Gradle from source. This is quite hard because Kotlin build depends on itself so we have to bootstrap it and this is a hard task. But help is wanted and we want Kotlin and Debian
04:00
so we like help. JVM languages, of course it is not all about Java. A lot of other languages run the Java virtual machine. We provide Groovy 2.14, Scala 2.11 2.12 requires SBT, that is Scala build tool.
04:21
Here we have some problems with 2. So help is wanted. We will chip closure 1.9, JITEN 1.71, JRuby question mark. It has currently three release critical bugs so I don't know if we can ship it and pass them. IDEs, unfortunately Eclipse is gone
04:40
because we have no maintainers who want to maintain it. We once had five, once. Today there is no one. I have packaged NetBeans 10 for Debian a few weeks ago which was a very demanding task but it will make it into Buster. Very important for server users
05:00
and we will ship with Jetty 9.4 and Tomcat 9 fully up to date and with systemd integration. Last but not least, reproducibility levels are 85%. What does it mean? So if we want in the future reach 100% that means you can verify if a binary package built in Debian
05:22
corresponds to the sources. So it gives you more confidence in what we ship and it gives you another security level or a new layer of security. You can learn more about the reproducible builds at reproduciblebuilds.org. This is a whole other talk, very interesting topic.
05:42
So now I want to talk about some challenges that we face in our daily package debug. You know that many build systems just download binary packages from the internet. This is the intended use case. So in Debian we say that our software requires
06:04
if the software requires other software from outside the main archive, that's not allowed. So we make sure, we promise you that everything in Debian is built from source and complies with the Debian free software guidelines and we cannot promise that if you can't control the binaries in external reports.
06:22
That's not possible. So every time someone ships only the binaries for example the end projects where you often can see that the developer includes jar files, then we have to remove them because we cannot control what's in them
06:41
and we don't have the sources. So how can we verify that this binary package actually corresponds to a specific source package? So there's security risks involved, there are software freedoms involved, so we say no downloads at build time, only build with packages that we have the source code and that we build ourselves, that we can control
07:04
and that will not vanish sometime in the future when some repo, an external repo will go offline or something else. Then the major point is Java is version centric. That means you have to pom file and you declare several dependencies.
07:23
So one project declares a dependency on a library 1.0. Then another project declares a dependency on the same library but version 2.0 and third one on 3.0 and so on. So for the casual Java developer that's not a problem
07:42
but that doesn't scale very well if you try to package that for a distribution like Debian. So we would have to package every version and include it with Debian and that's time consuming, ways of disk space, there's security risk involved.
08:03
What do we do if some security vulnerability is discovered? So you have to check is version 1 affected, is version 2 affected, is version 3 affected and then you have to update all these packages. This can work if you have a dedicated staff who works full time,
08:20
paid on your software projects and does nothing else than updating these projects. But it is very difficult for a volunteer project like Debian where you have unpaid contributors who have not that time. So a big problem is also that many projects ship
08:43
Uber jars or fetch jars or bundle everything into one big piece of software. I've seen even recommendations for shipping OpenJDK alongside with your application. So this opens enormous security holes.
09:00
It is very difficult to ensure that you can maintain this project. There can be a lot of security vulnerabilities. How can you verify that this will work in five years? So you have to constantly update the whole package and make sure that it works
09:21
and that's a very demanding task. So in Debian we say we ship only one version of the package. We make sure that this one version works with every other package in Debian and that you can independently update. This simplifies security support
09:43
and reduces code duplication but this use case is not very well supported upstream. So we constantly fight with build systems that assume it's okay to download various different versions of some library
10:01
and we have to patch them. And that makes life very difficult. This graph shows a little bit what we face. The amount of bugs that we fixed in the past years. You can see there is a slight increase
10:21
and obviously in 2018 something happened and you can also see that only a few contributors fixed a lot of bugs. I don't want to belittle those contributors that fix only a few bugs a year because we have more than a dozen different contributors but they concentrate on one package
10:42
and sometimes these packages are not affected by many bugs. So it is unfair to say they don't contribute anything meaningful. I would never say that. Or we have other contributors who just update documentation. So this is also very important but they won't show up here.
11:01
What I want to tell you with this graph is that we lack more people who want to get the bigger picture right. So we want to fix build systems, we want to fix bugs in libraries
11:21
which they actually don't care for but which are important for other packages. So it's not equally distributed. As you can see the yellow guy fixes almost three times more bugs than the next guy. And if he gets a cold or moves on
11:40
and does something else, yeah, we have lost a very important contributor and the project will suffer as a whole. That's obviously not good. Yeah, back to 2018. In 2018 the OpenJDK transitions happened, nine, 10 and 11.
12:00
And this caused a massive increase in bugs for example in build failures, most in build failures because classes were removed and methods were removed. The OpenJDK maintainers would say, okay, that was expected. We have deprecated all those classes years ago. Why didn't you fix it? The problem is many upstream projects say,
12:24
well, it works with OpenJDK 8, why should I update? But we have to make sure that we can ship OpenJDK 11 in our next stable release because that gets five years of security support. So if you ship OpenJDK 8 it will work now but not in five years.
12:41
So if you depend on security support on the server side then you have to choose a long-term support at OpenJDK. So it is not true that Debian is ships only an outdated software. Actually we work on the forefront but the actual development happens in Debian Unstable.
13:04
And if you download Debian Unstable you could see all this what's going on with the transitions and there's a lot of work involved to make every package buildable with OpenJDK 11 and also work at runtime with OpenJDK 11. I think it's a bit unappreciated by someone,
13:22
by some people, but it's a very important task for Debian. Yeah, that concludes my talk. Do we have any questions? Thank you very much. Anyone with a burning question? I will run to you with the microphone.
13:50
What do you do when you face a situation? So a piece of software is not compatible with OpenJDK 11. So how do you decide? Does your team act then and does something on the code
14:03
or do you reach out to the software project or are both approaches valid and how do you decide which approach to take? Thank you for the question. So there are usually three paths and paths. First of all we contact the upstream project
14:21
and look for new upstream versions, obviously. So we always try to have the latest software in Debian. So naturally that's the least amount of work for us. And if you discover upstream is not ready yet so we have the choice either to patch the actual version to work around the problem somehow or to actually fix it. If you can't fix it because it's a simple fix
14:43
or we just have the capabilities to do so then we forward the patch upstream and upstream will hopefully implement it and import it. If both of these ways don't succeed then we have to remove the software from Debian
15:02
because if you cannot build it from source then it doesn't meet our quality standards. Okay, thank you very much. Can I ask for a warm applause for Marcus? Thank you very much. Thanks a lot for the talk.