Being a good upstream - the syslog-ng PoV
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 90 | |
| Author | ||
| License | CC Attribution 2.0 Belgium: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/40263 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
FOSDEM 201376 / 90
2
5
8
10
12
13
14
15
17
19
21
24
25
28
29
31
32
34
36
39
40
43
44
46
50
51
52
54
55
57
58
62
65
66
67
78
79
87
88
00:00
Supersonic speedData managementProper mapComputer animation
00:32
Source codeComputer networkPay televisionRegulator geneProcess capability indexCodeMereologyMathematical analysisSoftwareProjective planeComputer programmingData storage deviceCore dumpWordConnectivity (graph theory)LoginEnterprise architectureProcess (computing)Data managementMessage passingDistribution (mathematics)TouchscreenDifferent (Kate Ryan album)Logic gateImplementationoutputFunction (mathematics)Open sourceLecture/ConferenceComputer animation
03:21
Public domainDefault (computer science)Revision controlDistribution (mathematics)Default (computer science)Source codePhysical systemSoftware developerExterior algebraInternetworkingEmailDivisorInteractive televisionElectronic mailing listProcess (computing)Meeting/Interview
06:55
Haar measureOnline helpDistribution (mathematics)Default (computer science)Patch (Unix)Different (Kate Ryan album)Computing platformRevision controlNumberSoftware maintenanceTouchscreenEnterprise architectureInstallation artLimit (category theory)Binary codeWebsiteInternet service providerComputer architectureRule of inferenceSource code
09:41
MereologyScripting languageDifferent (Kate Ryan album)Distribution (mathematics)Revision controlINTEGRALPhysical system1 (number)Lecture/Conference
10:49
Patch (Unix)Web pageSource codeModul <Datentyp>Distribution (mathematics)WikiProjective planeSource codeOnline helpRoundness (object)Patch (Unix)Different (Kate Ryan album)Denial-of-service attackoutputFeedbackInsertion lossSoftware developerCodeBitCore dumpLibrary (computing)SoftwareRevision controlModule (mathematics)Touch typingDatabase normalizationNetwork topologyNumberSoftware bugAnalytic continuationComputer animationLecture/Conference
17:00
Web pageSource codeCodeFingerprintGoogolDatabaseBlogNewton's law of universal gravitationRevision controlDifferent (Kate Ryan album)DatabaseDistribution (mathematics)Device driverSource codePatch (Unix)Software developerLibrary (computing)Projective planeFamilyMereologyCodeClosed setWeb 2.0Theory of relativityMessage passingPattern languageEmailParsingElectronic mailing listInformationInternet forumParsingBitComplete metric spaceSoftware bugSinc functionConfiguration spaceConnected spaceIP addressSearch engine (computing)Computer fileWeb pageGoodness of fit1 (number)Physical systemGoogolImplementationAddress spaceString (computer science)IntegerSoftwareGodStreaming mediaLoginComputer programmingExpressionFood energyMultiplication signFingerprintComputer animationLecture/Conference
25:51
Series (mathematics)Online help
26:52
BlogHill differential equationRevision controlDevice driverPatch (Unix)Food energyImplementationMathematicsSoftware developerPresentation of a groupPhysical systemDistribution (mathematics)Different (Kate Ryan album)Operator (mathematics)Projective planeNumberLecture/Conference
29:59
Computer animationLecture/Conference
Transcript: English(auto-generated)
00:19
Hello, I'm Peter Tonic from Barbit, I'm community manager for Syslogangy.
00:27
And a proper title would be that changing Syslogangy into a better upstream. First of all, I will talk about what is Syslogangy and the short history, then
00:45
about the programs it faced, and then I will also talk about how I tried to resolve these programs with the Syslogangy team. And at the end, I will
01:05
also talk a few words about our upstream projects. So what is Syslogangy? It's one of the most popular syslog implementations, often called a Swiss Army knife of logging.
01:28
It has a lot more possibilities than the original syslogd, a lot more inputs, outputs, better, much better filtering, and recently log processing
01:47
was also added. One can extract important data from log messages using Syslogangy. The Syslogangy code base is highly portable, running on just
02:02
about any Linux, BSD, and most commercial Unix variants. It's a software developed in Hungary, where I'm from, and developed for over 14 years as an
02:20
open source project. So Syslogangy is an upstream for quite a lot of different projects. It's a part of most Linux distributions, and it's also
02:43
available in BSDs in ports. It's the core component of enterprise log search and archive, which is a log management and analysis software. It's
03:03
also part of many networking appliances from Cisco, F5, or many network storage appliances. And it's also upstream for a commercial
03:23
version, which was added a few years ago. For many years, the Syslogangy community grown on its own. It was the only alternative to the good old
03:44
Syslogd, but knew a lot more, and was many more flexible. So it was documented over 50 books, and all over the Internet. And there were many
04:09
users of Syslogangy, but recently, many factors slowed down the adoption of Syslogangy. One of them is called R-Syslog. With the appearance
04:29
of R-Syslog, Syslogangy wasn't the only alternative anymore. Also, the introduction of the commercial version and the necessary contributing license
04:45
agreement alienated quite a few Linux distributions. Not the users, but as they receive, many of them use logging system from the distribution.
05:03
This also, it wasn't really good for us. And Syslogangy was never actively pushed from the developers. They just coded, answered emails on
05:24
the mailing list, and published source code. And that was all the interactivity. This had quite bad consequences for us, as R-Syslog
05:42
became the default in many Linux distributions, and also resulted in that many distributions stayed with ancient versions, Syslogangy versions,
06:02
as there was nobody actively working on this program. Three years ago, when I joined the Ballabid, I originally went there for
06:23
a completely different job, but I ended up to change the way how Syslogangy works and be a lot more active with the community. So,
06:48
before going on what I did to change the Syslogangy world, I would like
07:01
to ask a tricky question. What do you think? Which Syslogangy version is the most popular? Here are some hints on the screen. Syslogangy 3.4 was released just three days ago. SUSE Linux Enterprise has version 2.0,
07:25
Debian has 3.1, Fedora and OpenSUSE both have version 3.3. So, what do you think? Which has the largest install days? The first one
07:50
was the right answer, and even older, it's version 1.6, as Syslogangy is running on all of the Amazon Kindle devices. So,
08:04
over 50 million install days of Syslogangy. I guess no distribution can provide this number of installations, even with its default. So,
08:21
back to more serious topics. When Syslogangy team figured out that these rules are using ancient Syslogangy versions, they started to provide own binaries
08:43
on the Syslogangy website, which didn't really work out as expected, as one can build and provide the packages only for a limited number of distributions and architectures.
09:05
And just take Debian or NetBSD, where there are over a dozen different platforms, which we obviously cannot support on our own. So, one of the first things I did that I started
09:33
to look for active maintainers for Syslogangy packages in different distributions,
09:40
and have the work of these packages. This is beneficial for both sides. We know what is changed in Syslogangy. We know what are the dependencies, how they are changing,
10:00
what are the new features, and so on. But distribution packagers are the ones who know the packaging system, the init scripts, and the infrastructure of different distributions. So, this way we could
10:25
suddenly reach a lot more users, as it became integral, and up-to-date versions were
10:41
became integral part of different distributions. I also helped distributions to reinvent the wheel. I mean, many distros did quite similar patches to Syslogangy to help packaging
11:04
and stuff. And I started to collect these patches floating around and integrate them into our upstream tree. I also started to follow distribution backpackers,
11:31
as the Syslogangy project has its own bugzilla, but not everybody can find it. Most people find bugs at distribution backpackers, and this way we could collect a lot more
11:53
feedback and fix problems a lot quicker. Next, I would like to talk about how community
12:05
influences the development of Syslogangy. The first round is licensing and distribution policy. As I mentioned, the continuity license agreement was considered evil by a couple of distributions,
12:32
and it was quite well documented in different wiki pages, just nobody from the Syslogangy team
12:41
read it. So, one of my first input towards the team was that this problem should somehow be resolved, even if it might be too late and distros chose our syslog or
13:04
remained with syslogd, but at least we don't lose other developers and users and might gain even more if this obstacle is removed. So,
13:23
the first version after I joined the team was 3.2, in which Syslogangy became modular, which could quickly resolve this problem as the core of the software became lgpl,
13:48
the rest of the modules became gpl, which immediately made the CLA redundant, and if anybody looks at our git tree, the number of outside contributions immediately jumped.
14:18
We ran into another problem, the manual pages were not free enough, not a licensed group,
14:29
but it was something that was a problem for a very small distribution where all of the
14:43
code and documentation had to be under a gpl compatible license. We complied and did even more, we also made some XML sources for the manual pages available. Another problem is,
15:12
we run with ivykiss with the fedora project, we use ivykiss for threading and IO
15:27
in syslogangy 3.3, and we found ivykiss a very promising project, but there were many little details not yet worked out completely, so we fought the code base and
15:48
extended to our needs. We thought that we can wait with it and slowly upstream our changes, but as it turned out, while OpenSUSE, DBN, FreeBSD, and so on, did not care about
16:12
bundling for library, fedora was strict with us and did not allow to upgrade
16:29
syslogangy to version 3.3, so it became suddenly a huge problem for us,
16:43
about two-thirds of our users are coming from fedora that had CentOS and they couldn't use this way, our latest version from, sorry, from distribution
17:09
sources. So, we quickly, well not so quickly, but it took a few months, but we worked closely with the ivykiss
17:26
developer to upstream all of our patches and 3.3 is now part of fedora. Community can also
17:42
influence the development of a project with code contributions. Syslogangy tried to keep close relation with distributions in the past three years,
18:02
and once systemd became important, we started, the OpenSUSE guys approached us that they need support in syslogangy for systemd, and we did the initial
18:26
systemd support with OpenSUSE. Of course, systemd was changing quickly, and at least at the beginning, and we extended and finished our systemd support
18:49
with Arch Linux developers. There were many small features coming from the community,
19:00
pattern DB is the parser I mentioned which can extract information from log messages, and it has many different little parsers for IP addresses, email, and email addresses, strings,
19:20
integers, and so on, and many of these were added by the community. Our first larger external code contribution was a complete destination driver for AMQP.
19:47
We learned about it just by accident. Somebody was asking if we could find a bug in why syslogangy doesn't work in a sudden configuration, and we found that
20:10
there is an AMQP destination in the configuration file, so we asked if it could be upstreamed, and this is how it became our first large external contribution.
20:35
I mentioned that I'm working in close connection with many different distributions and Linux
20:43
distributions and BSDs, but of course there are a lot more Linux distributions than the major ones, and subscribing to backtracking and email list and so on is practically impossible.
21:12
One of the tools which can help to resolve this problem is a tool from Google, but I'm sure that
21:25
other search engines have similar. I guess the Big Brother is not the best expression for this, but practically this helps us quite a lot. Anytime syslogangy turns up on a new web page,
21:49
I get a message about it, and this helps to follow backpackers, email lists, forums, and so on, and collect information and also help where I can.
22:11
There are of course situations where the problem is very specific to software, like I cannot really
22:20
answer questions related to syslogangy integrated into F5 appliances, but often I just register to the given forum where something is found and try to help. And last but not least,
22:43
here is where Google has this Big Brother that I can also see what others are saying about syslogangy and how they compare syslogangy to competing solutions.
23:06
In the end, I would like to talk about our upstreams. I also I already mentioned Ivicus, but a bit contributed quite a lot of code upstream, and the Ivicus project is
23:23
our best upstream, and since version 3.3.6, syslogangy can work with upstream Ivicus. libdbi is the bad example for a bad upstream which we use to access
23:48
databases. The last release of libdbi was about five years ago. There are quite a lot of fixes in CVS, but as there was no release, distributions carry crappy old versions.
24:14
For example, syslogangy needs patches from CVS to access
24:22
I don't know, one of the SQL drivers. And for the Agri, I should mention MongoDB,
24:41
we were looking for C library which could be used by syslogangy, but there was just nothing what we could use, so we developed our own library for MongoDB, and it was
25:02
quickly picked up by other projects including rsyslog. At the end, I would like to ask if you have any questions.
25:30
Sorry? Yes, as far as I know, it was not me who contacted them,
25:45
that was a C++ implementation, but that didn't really fit our need, and
26:03
as far as I know, it was a colleague of me who tried to email with them, and we didn't really receive much support when asking for help, and it was easier to write something new.
26:26
As far as I know, there is now a new CVS solution, so we will look at it later on.
26:54
Any other questions? Sorry? Did you see there's a difference in how syslogangy is being received
27:16
by users and distributions since you made those changes? Yes,
27:25
the most important one I already mentioned, the number of outside contribution increased sharply, so for earlier versions, we only had very few patches from outside, and once
27:46
CLA was removed, just in a few months, there was a sharp increase in outside patches, and in a year, we got a really huge destination driver for syslogangy from the outside. We never
28:09
had this kind of contribution before, so it was a quite important and visible change.
28:28
The other one is, I was just sitting at Leonard's presentation about systemd, and previously, there was not really much outside cooperation,
28:47
just coding in the syslogangy team, and now we try to work closely with other projects, and Leonard emphasized that while other syslog implementations don't really care about Journer,
29:11
we will have a developer in the upcoming federal developers conference in two weeks, and helping to code jointly on syslogangy and the Journer. Sorry? Yes, thank you for that,
29:52
thank you for your attention.