Introducing OpenBSD's new httpd (part 1 of 2)
Formale Metadaten
Titel |
| |
Serientitel | ||
Anzahl der Teile | 41 | |
Autor | ||
Lizenz | CC-Namensnennung - Weitergabe unter gleichen Bedingungen 3.0 Unported: Sie dürfen das Werk bzw. den Inhalt zu jedem legalen und nicht-kommerziellen Zweck nutzen, verändern und in unveränderter oder veränderter Form vervielfältigen, verbreiten und öffentlich zugänglich machen, sofern Sie den Namen des Autors/Rechteinhabers in der von ihm festgelegten Weise nennen und das Werk bzw. diesen Inhalt auch in veränderter Form nur unter den Bedingungen dieser Lizenz weitergeben. | |
Identifikatoren | 10.5446/18667 (DOI) | |
Herausgeber | ||
Erscheinungsjahr | ||
Sprache |
Inhaltliche Metadaten
Fachgebiet | ||
Genre | ||
Abstract |
|
4
8
13
14
16
22
25
33
39
00:00
Bildgebendes VerfahrenCodeDatenverwaltungDynamisches SystemFormale SpracheImplementierungInformationMathematikSchaltnetzSoftwareTopologieZeichenketteFunktion <Mathematik>ProgrammierungTechnische OptikTermersetzungssystemProgrammbibliothekTypentheorieParserMakrobefehlHydrostatikBildschirmfensterGanze ZahlProgrammierumgebungTaskGrenzschichtablösungKonfiguration <Informatik>EntscheidungstheorieAggregatzustandArbeit <Physik>Arithmetisches MittelBetragsflächeBinärcodeBitCOMDAP <Computer>DatenloggerDivergente ReiheDynamisches RAMEinfach zusammenhängender RaumFormale GrammatikFunktionalGeradeHyperbelverfahrenInhalt <Mathematik>LastLokales MinimumMaßerweiterungMereologieMomentenproblemMultiplikationPaarvergleichPhysikalisches SystemProjektive EbeneRechenschieberSpeicherabzugSpeicherverwaltungTermWarteschlangeZahlenbereichFlächeninhaltKonfigurationsraumVersionsverwaltungGüte der AnpassungFamilie <Mathematik>Public-Key-KryptosystemServerExogene VariableAutomatische HandlungsplanungMatchingSpannweite <Stochastik>InternetworkingBasis <Mathematik>ParametersystemAusnahmebehandlungCASE <Informatik>Prozess <Informatik>VerzeichnisdienstWurzel <Mathematik>Coxeter-GruppeMetropolitan area networkUltraviolett-PhotoelektronenspektroskopieRoutingComputersicherheitRandomisierungVollständigkeitSpezialrechnerNabel <Mathematik>AdditionDämon <Informatik>PunktVererbungshierarchieKlasse <Mathematik>KontrollstrukturSchnittmengeEigentliche AbbildungOffene MengeAuthentifikationKernel <Informatik>Protokoll <Datenverarbeitungssystem>Wort <Informatik>Lesen <Datenverarbeitung>ChiffrierungWeb-SeiteArray <Informatik>ThreadInstallation <Informatik>BimodulBetriebsmittelverwaltungMailing-ListeDateiformatAliasingFramework <Informatik>GoogolLipschitz-StetigkeitSoundverarbeitungQuellcodeRegulärer Ausdruck <Textverarbeitung>PlastikkartePuffer <Netzplantechnik>Elektronische PublikationClientSichtenkonzeptWeg <Topologie>Web SiteSchlüsselverwaltungOnlinecommunityPatch <Software>Wrapper <Programmierung>Klassische PhysikSocketTLSKontextbezogenes SystemMultiplikationsoperatorSchlussregelFreewarePufferüberlaufExploitURLPlug inDifferenz <Mathematik>Schreiben <Datenverarbeitung>Message-PassingSocket-SchnittstelleRechter WinkelVirtueller ServerBenutzerbeteiligungInterface <Schaltung>OrtsoperatorMusterspracheComputerunterstützte ÜbersetzungDefaultTwitter <Softwareplattform>DokumentenserverSoftwareentwicklerEinsGeometrische FrustrationAttributierte GrammatikOrdnung <Mathematik>SoftwareTermersetzungssystemHalbleiterspeicherPASS <Programm>Physikalisches SystemResultanteVersionsverwaltungServerComputersicherheitDämon <Informatik>StrömungsrichtungAuthentifikationProtokoll <Datenverarbeitungssystem>Web-SeiteThreadInklusion <Mathematik>BimodulSkriptspracheRegulärer Ausdruck <Textverarbeitung>Elektronisches ForumSchreib-Lese-KopfAutorisierungPlug inBenutzerbeteiligungComputerunterstützte ÜbersetzungDefaultXMLComputeranimation
Transkript: English(automatisch erzeugt)
00:04
So, I think I kind of introduced me. I'm Reich. I'm a developer in the OpenBSD project for more than ten years now. I mostly like to work in the networking area and that's a lot of stuff there.
00:27
I think I have one comet in X as well. And yeah, actually for a living I'm running a company that does networking with OpenBSD.
00:45
But I didn't start working on OpenBSD because of the company, it was the other way around and so I'm in a lucky position that I can do what I like as my work, and we have a team
01:01
of a few people who also work in OpenBSD. So that's a fun part of it, but of course we also have to deal with customers and requests that are not really identical to the requests you have in the open source world. So, today I want to talk about HTTPD.
01:26
It is still fairly new. It showed up about a year ago and it's the new web server in OpenBSD.
01:46
HTTPD is included in OpenBSD since the 5.6 release. It was started just two weeks before the 5.6 release was finished and we decided let's
02:02
get it in because it's very new so it doesn't harm and so we had it in 5.6. But then it really matured in 5.7 which is relatively new.
02:21
5.7 was released in May. You have this nice Blues Brothers theme in 5.7. So buy CDs, go online, have a look where you can order it. That's supporting the OpenBSD project. So why do we need a web server in our base system?
02:40
Actually OpenBSD has a website and we want to serve the OpenBSD page which is in a very nice 1990s HTML layout still but we do need a web server for it to provide this page. We also have mirrors for the packages, the ISO images and so on and some of them actually
03:09
already switched to HTTPD because some of them are hosted in OpenBSD as well. Not all of the, hang on, people are tweeting me so I have to turn this off.
03:27
Not all of the OpenBSD mirrors are running on OpenBSD but actually many of them. So we do have a need for a web server in OpenBSD. But users maybe also want to use, set up OpenBSD and serve their own CAT page so they
03:46
can just install OpenBSD, run HTTPD and put their CAT pictures there. This is a real page that I found just by Googling it, the CAT GIF page. I think it's very nice.
04:04
And of course we want to serve it securely that nobody breaks in and put doc pictures there or something like that. We do have a looking glass for VGPD in our base system.
04:21
It's a simple CGI that I wrote some time ago and so it's not enabled by default but it's shipped with every OpenBSD relief. So just to provide a starting point, some exchange points are running VGPD and they
04:40
conveniently want to provide a looking glass usually to see what's going on, to do lookups and so on. For that we need a web server actually. Otherwise we would have to move this into ports but I like to have things on the base system actually. I rarely use ports except for like window manager and all that and the browser but for
05:07
the networking tools it's nice to have this in the base system in OpenBSD. So OpenBSD has a long history of web servers in the base system and the web server changed
05:20
a few times. So I give you a brief history. In 1998 OpenBSD introduced or imported Apache based on 1.3 release series I think or was it even 1.2.1 now it's 1.3.
05:43
I think Bob Beck did it. So OpenBSD 2.3 is like a long time ago. So it's very close still to the foundation of OpenBSD which happened I think in 95.
06:01
We're going to have our 20th birthday this year with the upcoming 5.8 release. So almost in the very beginning of OpenBSD we imported a web server. Apache 1.3 became old and we could not go to Apache 2 because Apache 2 has this Apache
06:26
2 license which does not fit in our licensing. It has some like weird requirement that would not work in OpenBSD. So we kept using Apache 1.3 and it became a fork mostly Henningbrouwer clean up the
06:49
Apache 1.3 and OpenBSD. He threw out stuff like optic support or VMS or something like that and we had
07:03
it hardened like doing change root by default and a few other things. So the OpenBSD Apache was quite different from the upstream version. In 2011 some people decided that Nginx is a cool thing now but actually Apache was
07:23
getting very old and there weren't any other requirements under a BSD license that were like small and nicely designed and Nginx was imported at this time. And then it took a while March 2014 actually last year when Apache was removed and Nginx
07:49
became the new default web server in OpenBSD. So last year in Ljubljana in Slovenia we had a general hackathon.
08:08
It really surprises me right now that it was last August because it's so far away but anyway so we were on this hackathon and we looked at the code base to replace
08:27
a few things to improve the security of our software and the base tree to use like better memory allocation and many other things.
08:42
I give more examples later and I looked at Nginx and it was not really easy to adopt our changes to Nginx without creating a big patch for it. So somehow I got frustrated and said well I wrote RelayD so RelayD is almost a web server
09:08
because it has some HTTP support and it does like all this asynchronous IO which is the nice part of Nginx and RelayD is doing this for a long time as well. So I sat down one day and stripped down RelayD, renamed the directory and removed everything
09:29
that is not needed like the health checking and so on and added support for serving files and at the same day I had a web server and so it happened that we decided to use
09:48
it instead of Nginx so Nginx had a very short time in OpenBSD actually. So in Japan I had a title like Security Shokunen but I think here I'm using a German term
10:06
like Sicherheitshandwirkskunst which basically means security craftsmanship in German. As I heard that you like our long words. So we constantly improve our code base for security and quality that's the nice thing
10:25
in OpenBSD, it's not just like a graveyard of code. Something that is in the base system is something that's supposed to be reviewed and modified to have like a common thing.
10:41
If we introduce a new security API or the allocator or somewhere else we go through the tree and adopt it everywhere so all the time. And then last year all these things like Heartbleed and Shellshock happened
11:04
and one response was to create the LibreSSL fork basically. I was kind of involved in that, I was a messenger. I talked to CEO and CEO said yeah sure convince the people to do it and I did and then
11:26
it happened so I convinced other developers actually I'm not so active in the development of LibreSSL but at least I had the messenger role and I'm still alive. So in the reaction to that we also introduced like realloc array for example that's one
11:50
function that is supposed to reply unsafe array allocations where you do like you want to allocate an array and then you write in for example calloc with n times m in it
12:09
and these area locations are possibly vulnerable to overflows and realloc array allocations array is a new function we have an open basis that does a bounce shaking internally
12:22
so if you allocate an array that none of the values would overflow the integer basically. So it's a protection against some attacks that happened and I tried to adopt this to nginx because nginx allocates pools and arrays all over the place and they just
12:43
assume that the kernel will always give you like values that cannot overflow and like these it is safe we can just safely assume it is something we don't really like to do we want to explicitly check is there an overflow or not and not say no this cannot happen so I tried to apply it to nginx and the diff got big and we couldn't get it upstream
13:08
so we did not want to maintain it in open bsd ourselves so it just I throw away the work and we intended to use nginx as it is.
13:24
That's one tweet I wrote the next day after I wrote httpd and at the same day well very late the day Bob Beck and Theo Derratt gave me some beer and said okay can you import
13:43
this web server into open bsd I was scared I mean it was just new and so everyone knows that as a developer writing a web server is like what everyone does who learns programming language web server is like the hello world of networking tools so you don't really
14:03
do it so I wrote this is the server and then suddenly see you on that Bob are pushing me to get it in the tree and the beer helped so next day I woke up and realized that I had committed a web server so.
14:26
In the beginning we had httpd it was not yet enabled I worked on it for like two weeks in an insane run basically just me after the hackathon I went home and I didn't do
14:41
any other work and my family didn't really see me so I had this two weeks when I got the web server in a state that it was usable there were some issues though but it was usable for basic set ups already and so Theo said okay so we enable it so first
15:03
we import stuff in the tree but it's not linked to the build it sits there with a make file but then when enabling it it gets compiled and it becomes part of the snapshots and releases so in 56 it showed up actually.
15:20
We had TLS support contributed by Joel saying the basic file serving and fast CGI was contributed by Florian OBSO everything within these two weeks but of course we continued working on it this is not the current state so the design simplicity is a goal httpd is designed
15:46
to be a simple and secure web server I mean maybe these days everyone claims to be secure and simple but then I did some research looking at other servers and none of them
16:00
really satisfied me so it's not that I really wanted to write my own it's like the frustration was with others engine X for example started fairly small but more features got added over the time and vendors and all these influence so it's not simple
16:21
anymore it's quite big and in other ones or even light years it's not simple anymore it's not light anymore so httpd should remain simple have like the basic task to serve static files do fast CGI for dynamic content do proper TLS like securely and some other
16:50
core features should be built in directory listing of course logging basic authentication so the current code is allowed of 11k that's from current to the current version of the
17:09
current actually can you can you read this or is it too light I don't know so the different files including the documentation and the or the man page and the make file so it's
17:24
not big actually it's the task was not to write the smallest web server possible this design includes like privilege separation and and and proper and and and and and design actually so it's not just I write a web server in one file it's it's solid actually
17:47
so for for that what it does it's fairly small a few features so of course it does static files and directories then we do support fast CGI it is secure by design for example
18:05
in open bsd we had to patch Apache the web server to run in a change route by default I'm not sure if anyone is doing this by default in open bsd we're doing it for years so the web server is dropping privilege and change routes to an open bsd slash bars
18:26
like dap dap dap so in open bsd shell shock is not possible by design unless you copy a shell binary to to the the web server route so accessing like etc or etc as I learned
18:48
files is not possible with a change-routed web server and in most cases this is totally fine we had this patch for nginx for some time and for some reason it didn't get
19:01
accepted as well but okay fine we are used to that we we maintain it for nginx ourselves but HTTP is the first web server that I know of that is designed to be change route so you cannot turn it off if you need to access etc then then you can change route to slash
19:20
maybe but it is not intended to be yeah unchanged route or something like that um it's doing more than change route it's doing privilege separation I will show this later TLS is there of course specifically for libre SSL you might be able to compile it
19:45
with open SSL but some of the API extensions that we have in libre SSL are used by HTTP so HTTP is really like the reference implementation for a TLS library I talk about this later virtual servers of course reconfiguration on the fly so
20:11
you don't have to kill and restart you can just reload the configuration by keeping it running logging via syslog or files of course you don't have to buy a pro version
20:25
to do syslog logging it's integrated actually we have some basic rules to block and drop connections and then a user contributed support for streaming so byte ranges actually
20:44
it's a really nice thing that happened not so long ago so byte ranges will be in 58 it's not yet at 57 then I have something I think unique I have this pink label in
21:04
github I use github not for the development the development is happening in openbst cvs I use github for the issue tracking so in the issue tracker you can create labels for like won't fix and whatever and I created a labor feature riders to mark feature
21:23
requests from users that are out of our scope just to remind us that this feature is not intended to be an HTTP and then if anyone shows up and asks for that feature again I can simply point of it actually the user community learned it very quickly that could
21:44
you add this feature or is it considered to be feature riders so I think it's a really good thing that people get an awareness that not every feature is going to be in the software so tracking the things that we are not going to implement the not to do list is I think
22:03
something really nice and it works really well on the other hand there's hope some of the requests are rejected now but maybe I change my mind at some point right just to have like a future release I was thinking about Apple like a major feature is missing
22:23
in the initial release and I say no way and then maybe in a year it shows up and everyone is excited again so but what we not planning to implement is other cgi interfaces in addition to fast cgi and people are having long arguments without why why uwsgi is so much better
22:49
with python and you have this other frame work and blah blah blah but actually normally you can use fast cgi and the implementation in HTTP or fast cgi is actually very fast
23:02
it's not writing the output of the cgi to a temporary file to serve it to the internet it's streaming it directly so it doesn't make sense for us to add multiple latest and greatest cgi protocols and for authentication we do support basic authentication
23:25
but there is no plans to add support for ldap or something like that if you need to then install nginx from ports nginx is still really powerful and a good software and so for advanced use cases it's still in our ports tree for the basic things
23:44
HTTP is probably the preferred option in open bsd already we don't support modules plugins HTTP 2 support yeah I agree that's one of the rare cases when I agree with phk
24:00
he he wrote something in the ACM queue I think about HTTP 2 and why he's not going to support it in varnish and the protocol is insane actually so I don't know some people want it somehow or it would probably make sense in relaydee to do HTTP 2 to HTTP 1
24:24
relaying or something like that and or asynchronous design allows HTTP 2 support but it's madness so I don't know I have no convincing arguments to to implement it or and we are not going to support regular expressions that's what people are writing about but I'm not
24:45
doing it but so rewrites are not possible security it runs change rooted by default as I said it uses privileged separation so three processes the parent that loads
25:07
the configuration open socket loads keys and all that the the server handles the HTTP connects we can have multiple server processes and the logger is an extra process for logging
25:23
we try also from a design point of view don't reinvent the the wheel don't use our own string API we use lip C whenever possible even if there is like a possible minor performance trade-off I prefer to use lip C functions for example in engine X optimized HTTP parser
25:52
there are like individual string comparison functions depending on the number of arguments or I don't quite remember the names but there's a str com for five characters and then there's
26:09
our str com for four characters and it's like it's super optimized and it's very fast and but in OpenBSD we like to use our lip C because then we can tweak something
26:24
in our default libraries and everything benefits from it and we don't have to look into all these specific places as we know from OpenSSL that's actually also a good idea OpenSSL use its own memory allocator I think it's probably still doing that but we threw it
26:44
out in LibreSSL LibreSSL is using the system malloc so LibreSSL is not doing the exploit mitigation mitigation anymore that OpenSSL used to do actually it surprised me
27:01
a lot that a few months after we we did this in OpenSSL and removed it from from the LibreFog that I found all these custom allocators in the other web server so okay that's a design decision for performance it makes sense there but we don't want that we want our hardened malloc that does randomization and use after free detection and so on
27:30
so the privilege separation are really processes that communicate with each other the parent forks them in the beginning and then they just run there's no respawning or something
27:43
like that you can configure the the number of server processes and then each server process handles the the connection with asynchronous IO so there's no threading involved or something like that and the server processes for example don't have write access to the log
28:04
files they send a message to the log process basically the nice side effect is you can have multiple server process and the messages to the single logger gets serialized because of the messaging and the performance is still really good so we we can open log files
28:29
with the right privileges that are compatible to the other web servers but the the server processes don't have to touch them and there are some other things so the server
28:40
process for example they also run with an unprivileged user they they they cannot do anything harmful and if we ever need another thing we might add another privilege process and relay D for example we have another process for the RSA private keys and open SMTPD
29:04
I didn't add that to HTTP yet but I will at some point so when we did it in libTLS so LibR ssl added a new API on top of libssl it in the beginning it was called
29:22
libre ssl but this was a quite confusing because libre ssl sounds like libre ssl but but this is actually a part of libre ssl so now it's called libtls simply and it's basically an API on top of it but it's so easy to use you should really have a look at it
29:44
you can write TLS clients or servers in just a few lines and it does everything right so Joel thing is doing the the major work there and I'm doing it from a reference implementation point of view so
30:05
in HTTP we decided instead of using libssl directly the old API that you know from open SSL we we use libTLS so this also helps to shrink the size of HTTP
30:22
and by default it only does TLS 1.2 for some months now and only strong ciphers and so on so logjam for example wasn't an issue for HTTP fast CGI as I said was contributed by Florian Opse another German I asked him can you give me a quote for the presentation
30:45
why did you implement fast CGI and he said I implemented slow CGI that was the CGI wrapper that we had before I implemented slow CGI because you didn't stop whining on ICB that nginx can't
31:00
execute BGP-LG and fast CGI in HTTP Bob has asked me if I can help you with it so a little bit back when we removed Apache there was no run to run the BGP looking glass anymore because it is a classic CGI and nginx is not supporting the classic CGI interface
31:25
which is the right way to do so we needed a fast CGI support in the BGP looking glass or a fast CGI wrapper so Florian showed up and wrote this slow CGI which is basically
31:40
a little server that helps you to run traditional CGIs and then talks with fast CGI to the web server and he used this code later because it's a new implementation of the fast CGI protocol without depending on the official libraries and all this bloat so he used this to
32:03
to write the fast CGI server code for HTTP which works really well and we do as I said direct streaming there's no intermediate buffering to a file the configuration that's also an example I hope you can read it the next slide I will give you an example of the
32:25
basic web server configuration so you open a text file and put that in the text file httpd.conf and then it's working okay that's all you need actually I'm thinking about making
32:47
the listen on port 80 as a default as why so you can run it with an empty file or something like that but that's a minimum requirement so yes yes yes yeah we don't do reg reg x
33:09
but we at the moment we do support the fm match globing rules so you can do like shell shell white cards basically what people also do is like you do star.example.com and as
33:27
virtual hosts
33:41
then manual page displayed with man CGI including an online source code repository where you can look at history the complete configuration file and 66 lines with httpd and I'm writing that since November last year so he's what he's saying is really accurate yeah so and since then we
34:07
even added like name-based aliases and all that that helped to reduce it further in my case so that's a bit more advanced for example you can include an external mime types file
34:23
if you don't do it it provides a list of the most common types like html jpeg javascript and just otherwise you can just use the existing mind of types files compatible to the apache slash engine x format there for that we even for the mind types we even pass these
34:45
semicolons at the end of the line because as you see we don't need semicolons at the end of why the grammar is using the same parser that pfctl does the path dot y pump pf or bgpd relay d we use it in
35:04
in many places in open bsd right now that's our unified configuration actually without breaking ls um we we can anyway so no without using an external library or something like this and obviously we just
35:22
reuse this past of y code that originated from ps parser and then we we use it in all the other new demons bgpd anti pd even relay d all of them and so you can use macros like in pf
35:43
you don't have to write semicolons at the end of the line and it's very similar some advanced configuration it's very bright anyway um so you can listen on multiple ports you can also add additional server names for
36:07
for name-based aliases logging is enabled by default but can turn it off locations is the matching is also using fn match at the moment as i said we're not going to do red
36:22
regular expressions so there are a few options they're all documented on the mount page as usual in open bsd i think the mount page is in really good shape so that you can understand what it's doing and it's not like long and you don't have to pick it from the web page or so it's just do manage the pd.conf blocking rules are supportive or for redirections you can redirect
36:49
and so on fast cgi a few other options it works well in combination with php fpm of course but also with with many other frameworks future work that is uh
37:10
that's very new actually not even all of the open bsd developers know about it because it hasn't been released yet ceo is working together with me and a few other developers
37:26
the most of the work was done by i forgot his name was it nick m and you will figure out when it's released so somebody in open bsd implement something ceo is designing it we we're working
37:40
on a new framework to improve privilege separation and to further draw privileges but it's designed in a way that it's practical it's a practical approach it's easy to use so basically the kernel limits the the interfaces to a substance that of of the posix and the
38:06
environment that you need in the individual process and it works really well with privilege separation for example ht ht pds logger process doesn't have to open any network sockets so we can we have a class basically that we can drop it's much easier and better designed than like
38:28
cyst race for example or the the other things in other systems it's not trying to solve every possible problem it's trying to be a practical practical approach so stay tuned actually it will
38:43
be really nice and we will use it everywhere actually so more features are in preparation like the sni support is i promised it before but it will come rewrites well
39:03
not with uh regular expressions but we found a very nice way and that is currently being investigated so we can do rewrites and advanced matching but with a matching language that that
39:21
i can understand where i can read the source code and know what's going on i think i asked michael lucas what do you think about regular expressions and he said oh people are asking me all the time to write a book about reg x but why do we have to write a book about it so when it's so complicated in the first place and i don't want to use something in ht pd just
39:43
for the pattern matching where you have to read books and books to get it right so we found something else and i hope that i can release more information about soon but actually i just started looking at it yesterday so yeah this tami i think tame in english i think in in japanese
40:10
it also has a nice meaning so um so tame uh will limit the privileges of each process so you can decide that the server process is not able to i don't know do change the the system
40:30
time or for the logger i think the example is good that it doesn't have to open any network sockets and so on so this uh is once again very easy to use and it will further improve the
40:43
security of ht pd but it's not specifically for ht pd mostly everything in base will use it yeah so open bc57 was released in may buy the cds support the project and have a look at the
41:03
funding campaign for this year and buy off beer actually so thank you