Introducing OpenBSD's new httpd (part 1 of 2)

Video thumbnail (Frame 0) Video thumbnail (Frame 2597) Video thumbnail (Frame 5437) Video thumbnail (Frame 6332) Video thumbnail (Frame 7808) Video thumbnail (Frame 19756) Video thumbnail (Frame 31447) Video thumbnail (Frame 43827) Video thumbnail (Frame 45815)
Video in TIB AV-Portal: Introducing OpenBSD's new httpd (part 1 of 2)

Formal Metadata

Introducing OpenBSD's new httpd (part 1 of 2)
Title of Series
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.
Release Date

Content Metadata

Subject Area
OpenBSD includes a new web server in its base system that is based on relayd and replaced nginx. OpenBSD includes a brand new web server that was started just two weeks before the 5.6 release was finished. Work is in active progress and significant improvements have been done since its initial appearance. But why do we need another web server? This talk is about the history, design and implementation of the new httpd(8). About 17 years ago, OpenBSD first imported the Apache web server into its base system. It got cleaned up and improved and patched to drop privileges and to chroot itself by default. But years of struggle with the growing codebase, upstream, and the inacceptable disaster of Apache 2 left OpenBSD with an unintended fork of the ageing Apache 1.3.29 for many years. When nginx came up, it promised a much better alternative of a popular, modern web server with a suitable BSD license and a superior design. It was patched to drop privileges and to chroot itself by default and eventually replaced Apache as OpenBSD's default web server. But history repeated itself: a growing codebase, struggle with upstream and the direction of its newly formed commercial entity created a discontent among many developers. Until one day at OpenBSD's g2k14 Hackathon in Slovenia, I experimented with relayd and turned it into a simple web server. A chain of events that were supported by Bob Beck and Theo de Raadt turned it into a serious project that eventually replaced nginx as the new default. It was quickly adopted by many users: "OpenBSD httpd" was born, a simple and secure web server for static files, FastCGI and LibreSSL-powered TLS. And, of course, "httpd is web scale".
Area Web 2.0 Server (computing) Computer animation Software Open source Software developer Right angle Open set Whiteboard Mereology Position operator
Web page Metropolitan area network Server (computing) Server (computing) Web page Set (mathematics) Control flow Open set Open set Storage area network Web 2.0 Medical imaging Order (biology) Computer animation Website output Musical ensemble Gamma function Computer-assisted translation Family Mathematical optimization Physical system World Wide Web Consortium
Point (geometry) Metropolitan area network PC Card Default (computer science) Server (computing) Computer-generated imagery Server (computing) Multiplication sign Maxima and minima Basis <Mathematik> Open set Web 2.0 Data management Computer animation Internetworking Traffic reporting Window Physical system Exception handling
Wechselseitige Information Group action Context awareness Dynamical system Code State of matter Multiplication sign Range (statistics) 1 (number) Set (mathematics) Frustration Open set Mereology Computer programming Usability Web 2.0 Fluid statics Mathematics Web service Virtual reality Semiconductor memory Core dump Extension (kinesiology) Resource allocation Information security Physical system Area Metropolitan area network Logical constant Computer-generated imagery Software developer Computer file Electronic mailing list Internet service provider Drop (liquid) Bit Flow separation Open set Connected space Self-organization Right angle Summierbarkeit Block (periodic table) Curve fitting Row (database) Point (geometry) Web page Functional (mathematics) Server (computing) Sine Implementation Regulärer Ausdruck <Textverarbeitung> Open source Computer file Dependent and independent variables Patch (Unix) Ultraviolet photoelectron spectroscopy Directory service Automorphism Rule of inference Gezeitenkraft Intranet Root Robotics Term (mathematics) Natural number Software Gastropod shell Directed set Integer Interrupt <Informatik> Gamma function Task (computing) World Wide Web Consortium Authentication Default (computer science) Dependent and independent variables Server (computing) Forcing (mathematics) Graph (mathematics) Content (media) Code Directory service Line (geometry) Uniform resource locator Word Kernel (computing) Computer animation Software Personal digital assistant Network topology Data Encryption Standard Natural language Family Buffer overflow Local ring
Context awareness Randomization Structural load Logarithm View (database) Modal logic Decision theory Source code Set (mathematics) Bit rate Function (mathematics) Parameter (computer programming) Open set Client (computing) Hyperbolic function Mereology Web 2.0 Mathematics Semiconductor memory Information security Trail Computer-generated imagery Structural load Software developer Computer file Electronic mailing list Port scanner Regulärer Ausdruck <Textverarbeitung> Public-key cryptography Flow separation Open set Connected space Message passing Process (computing) Configuration space Right angle Modul <Datentyp> Point (geometry) Onlinecommunity Server (computing) Functional (mathematics) Implementation Computer file Wage labour Transport Layer Security Number Element (mathematics) Goodness of fit Flow separation Read-only memory Internetworking String (computer science) Implementation Mathematical optimization Authentication Module (mathematics) Socket-Schnittstelle Addition Default (computer science) Pairwise comparison Multiplication Key (cryptography) Inheritance (object-oriented programming) Server (computing) Interface (computing) Cellular automaton Planning Line (geometry) Exploit (computer security) Spring (hydrology) Computer animation Software Personal digital assistant Network topology Key (cryptography) Communications protocol Library (computing)
Demon Building Presentation of a group Code Ferry Corsten Multiplication sign Source code Combinational logic Coma Berenices Shape (magazine) Mereology Computer programming Formal language Web 2.0 Computer configuration Software framework Office suite Information security Physical system Social class God Metropolitan area network Computer-generated imagery Wrapper (data mining) Software developer Moment (mathematics) Electronic mailing list Virtualization Maxima and minima Bit Regulärer Ausdruck <Textverarbeitung> Flow separation Open set Type theory Arithmetic mean Process (computing) Buffer solution Configuration space Pattern language Right angle Quicksort Reading (process) Row (database) Aliasing Classical physics Web page Slide rule Game controller Server (computing) Existence Implementation Socket-Schnittstelle Computer file Software developer Expert system Regular graph Product (business) Integrated development environment Macro (computer science) Form (programming) Socket-Schnittstelle Default (computer science) Addition Matching (graph theory) Information Server (computing) Interface (computing) Projective plane Computer network Line (geometry) Word Uniform resource locator Kernel (computing) Computer animation Integrated development environment Software Personal digital assistant Formal grammar Natural language Game theory Communications protocol Library (computing)
so I think I'm kind of introduced me I'm I'm right I'm a developer and the obviously projects and for more than 10 years now I mostly like to work in the networking area and that's a lot of stuff there I think I have 1 comment in excess of the world and so on and yet actually for a living I'm running a company that does networking was opened in the but I didn't start working in open is the scores of the company it was the other way around and so I'm not lucky position that I can do what I like about my work and we have a team of a few people who also work in the area but that's the fun part of it but of course you also have to deal with customers and requested or or not really the identical to the request to have in the open-source world so today I want to talk about http the it is still fairly new and it showed up on board a year ago and that the new web server and open with the issue
and it is included in all 7th the 5 to 6 release it was started just 2 weeks before the 5 6 release for Finnish and we decided or let's let's get it in because it's in it's very new so of almond because we really hadn't 5 6 but then it really majored in 5 7 which is a relatively new 7 released in and may made you have this nice blues brothers 1st so by CDs go online and look where you can order it that's supporting your previous approach so why do we need a web server and all based system actually obviously have a website and we want to serve the opening page which is in a very nice the nineteen-nineties HTML I hold still but but we do need a web server for it to to provide page we also have minerals for uh the the packages the eyes of images and so on and some of them actually already switched to ht http because some of them are are hosted and obviously as well not not all the same thing on people are treating me so as to turn off of that had not not all of the optimism Mayrhofer running on a previous but at the many of them so we do have a need for a web server in open but but but users
may be also want to use set of old music the and their own cat page so they can just insulting B is the run http and put the cat pictures there and this is the real page that I found just by googling at the cat give page I think it's a very nice and of course you what we want to a servant securely that nobody breaks and input stock pictures there or something like that
we do have looking glass for a the GPD and are based system and it's a simple CGI there so that I wrote some time ago and so it's not enabled by default but it's should with every obviously relieved so just to provide a starting point some exchange 1 surrounding the GPD and and their conveniently 1 to provide a looking glass you lead to see what's going on to look up and um for that we we need a web server otherwise we would have to move this isn't reports but I I like to have a single base system actually I I rarely use ports except for like window manager and all that and the the browser but for the internet talking to it's it's nice to have this on the basis of an optimist so obviously
has a long history of web servers in the base system and the web so what changed a few times so they give you a brief history in
1998 and open this introduced or imported Apache and based on 1 ancestry release serious a single was it in 1 2 . 1 9 1 . 3 the simple fact that it's so openly the tool points 3 is like a long time ago it's very close to to to the you know the foundation of obviously which happened to sing in 95 we going to have our 20 is birthday this year with the upcoming 5 8 released so I almost in the very beginning of obviously the we imported the web so an Apache among points really became old and and we could not go to Apache 2 because Apache to hess's Apache 2 license which does not fit in our licensing and the that it has some like we're requirement that would not work and open so we we kept using Apache 1 3 and it became a for the some most the hand and brought clean up the Apache 1 3 and open B is the sum of his throughout stuff like big support and the Guinness or something at the end we we had at a hardened like doing and changes by default and a few other things so the obvious the Apache was quite different from the upstream action in 2011 some people decided that engine X is a cool thing now what actually Apache was getting very old and there aren't any other requirements under realizing the word like small and so nicely designed and and was important at this time and then it took a while March 2014 actually like last year when an Apache was removed and then the next week on there and the new default web server and all this so last year in some may be Ljubljana in Slovenia we hadn't had a general Hackett it really it surprises me right now that it was lost or because so far away but anyway so that we're the 2nd and so did we we looked at the Government Code Base to to replace a few things to to and improve the security of foreign of softer in the base treated to use like better of memory a location and and and many other things that give more examples later and I looked at and genetics and it was not really easy to adopt or changes to genetics result creating a big patch for so somehow I got frustrated and said well I I wrote we lady 3 lady is almost a web server because it has some http support and and of like all this as synchronous I O which is the nice part of the and genetics and related he is doing this for a long time of work so I said on Monday and some stripped down lady we named the directory and removed everything that is not needed like the hell checking and so on and added support for storing files and at the same day and I had a web service and so it happened that we decided to use it instead of excellent annexed had a very short time and obviously actually so in in Japan and I had a title like security should Koonin but I think here I am using and German term at the shall heights context consists which basically means security craftsmanship in German and as you like all known words so so we constantly improve our code base for security and quality that's the nice thing and obviously it's not just like with a graveyard of code something that is in the base system as something that's supposed to be reviewed and um modified to to to have like like a common thing but we we if we introduce a new Security API layer LOB LOD somewhere else we we go down through the tree and adopted everywhere so all all the time and then last year all these same like hot lead and and show shock happened and 1 response was to to create leverage out for basically so I was kind of involved in that was a messenger at the top is unity is that yeah sure convince the people to do it I did it and then it happened so I convinced that other developers actually I'm I'm not so active in the development of liberal society but at least at the messenger organ and I'm still alive so so in reaction to that we also introduced like we are local area for example that's 1 some function that is supposed to reply an unsafe area locations where you do like that you want to allocate an area and right and for example see on a quiz n times m in it and and these area locations our possibly 1 robot to overflows and we are local area so new function we have an open this is that that of the bulge shaking interim so if you allocate an area that none of the value of would also the the integer and so it's a protection against us some attacks that happened and I tried to set up this 2 enter next because and next allocates pools in areas all over the place and they just assume that the kernel will always give you like value of that cannot overflow and like these it is safe we can just safely assume it is something we don't really like to do we want to explicitly check if they're an overflow on automatic now this cannot happen so I tried to apply to intranets and that is what Bacon but we couldn't get it upstream so we did not want to maintain and an open this yourself so
it just I throw away the working 3 you intended to use genetics and evidence so that's 1 to read I wrote at the next day after and I wrote http the and at the same that a day will very late in the day at Bob bank ensued Iraq gave me some beer and said OK can you importers web server interlocking this segment but I was scared I mean it was just new and everyone knows that as a developer writing a web server is like would anyone that's a learning programming that miniature web service like the hello world of networking tools so you don't really do it so I wrote this as a server and then suddenly you see on the ballpark pushing me to get in the tree and the Buick helped so next day I woke up and realized that I committed to the web server so I in the beginning we had http it was not yet enabled and worked on it for like 2 weeks in an insane are run basically just me and after the Hackett town I went home and I didn't do any other work of my and my family's been released see me so I had to to really explain I I got the web server in in in in in a state that it was usable there were some issues still but it was usable for basic set ups already and soul to use that OK so we enable so 1st we import stuff in the tree but it's not linked to the build it said there was a Makefile but then when enabling it it gets compiled and it becomes part of the snapshots and release so and I think that showed up actually I we had the last support contributed by Joel thing the basic 5 serving and Fast CGI was contributed by Florian also everything was in these 2 weeks but of course we continued working on it this is not the current state so that the sine simplicity usable and HTTP is designed to be simple and secure web so relying maybe these days everyone claims to be secure and simple but then I did some research looking at other services and none of them really satisfied I mean it's it's not that I really wanted to write my own it's like the frustration was with others and annex for example started fairly small but more features added over the time and then and all these influence so it's not simple anymore it's quite a bit and then other once or even like to use this not simple anymore it's not like anymore so HE DPT should remain simple have like the basic task to serve static fights do Fast CGI for dynamic content to property unless like securely and do some of the core features so that should be built in directory listing of course lobbying Basic authentication so the current code is a lot of health at 11 K that's from current actually and you can you read to sort the to light and so these these different files including the documentation and the ornamentation the file so it's not big picture it's the task was not to write the smallest max over possible this design includes like privilege separation and to implement proper and design actually so it's not just I write red so 1 1 file it's it's it's solid actually so for all that what it does it's fairly small if you features so of course Adele static files and directories then we do support fast G. I it is secured by the lines for example in open B is the we had to patch Apache the web server to run a change through by default and natural anyone is doing this by default in all the 3 doing it for years so the web server is dropping privilege and change roads tool in open because the flesh like Dr. . so in Albany the shell shock is not possible by design on if you copy it shall binary 2 2 0 the web server over root but so accessing like ETC or at sea learned of files is not possible with the change of graphs and in most cases this is totally fine we had page genetics for some time and for some reason it didn't get accepted as well of key find very used to that we we maintained at 4 and Annex ourselves and about http is the 1st web server or that I know of that is designed to be changed to you cannot turn it off if you need to access ETC then then you can change through to slash maybe but it is not intended to be a yeah and and change root of something and it's doing more and then changes the thing privilege separation of world show this later TLS there of course specifically fully you might be able to compile a because of missus but some of the API extensions that we have a new versus are used pay http decent http is really like the reference implementation for last 1 of the they talk about the which also rows of course of reconfiguration on the fly so and you don't have to kill and restart can just ought to consideration were kidnaped running lobbying biases slorc or only files of course you don't have to buy a pro worse introduces look lobbying it's integrated to actually and you we have some basic rules to to block and dropped connections and then add a user contributed but support force streaming so byte ranges which really nice seeing that happened not so long ago also byte ranges will be in 5 8 it's not that of 5 7 then I have
something I think unique I have pain label in and detail by user did up not for the development the development is happening in optimistic CBS used get up for the initial attracted so in the issue tracker you can create labels for like won't fix and whatever and I created a labor of future writers and to market feature requests from users there are all Horrorscope just to remind us that this feature is not intended to be an HTTP and then have any 1 shows up and ask for that future again I can simply point of actually the user community learned very quickly they all could you add this feature is considered to be the future right so I think it's a really good thing that people get an awareness that not every future is going to be in the self and soul tracking the things that we are not going to implement the not to do list this sing something really nice and it works really well on the other hand there there's hope of some of the requests are rejected null but maybe I change my mind at some point but just to have like a future release I I was thinking about apple like the a major is missing in the initial release necessary in no way and then maybe in a year it shows up and everyone's excited but would really not planning to implement this other CGI interfaces in addition to Fast CGI and people are having long arguments without why why you WSG I so much better was Tyson and you have this other framework and blah-blah-blah but actually normally you can use Fast CGI and the implementation in http via Fast CGI is actually very fast it's not writing the output of the CGI to a temporary file to the server to the internet is screaming at directly so all of this and it doesn't make sense for us to at at multiple latest and greatest CGI protocols and fault occasion we do support basic authentication but there is no plans to support for L . 4 something like that if you need to then install and annexed from ports and annexes still really powerful and a good software and so for advanced use cases it's still in all ports tree for the basic settings http the is probably the preferred option open or we don't support modules plotted against http to support me yeah I agree that's 1 of the rare cases when I agree with pH K he he he wrote something in the ACM Qs think about http to and why it is not going to support it in varnish and the protocols insane actually so I don't know some people wanted some whole would probably make sense in the lady to do http to to http 1 relaying or something like that and or synchronous design allows http to support but it's madness 0 and I have no convincing arguments to to implement or and we are not going to support regular expressions from that's what people are whining about that we but not doing it but so we rights are not possible security it runs change through the body fall and it uses privilege separation so 3 processes that parents that loads the configuration open soccer loads keys and all their their this over handled the http connects with the kind of multiple so processes and the longer as an extra process for logging and we try also from a design point of view don't reinvent the wheel don't use all spring API we use Lipsey whenever possible even if there's like a possible minor or performance trade-off but I prefer to use that C functions for example in an engine x optimized http parts they're like string comparison functions depending on the number of arguments a don't quite remember all the names but there's str come fall of firefly character and then they're thrown if there are come for for character and it's like it's super optimized and that's that's very fast and but in open this the we liked to use lips see because then we can tweak something in our default libraries and everything benefits from it and we don't have to look into all the specific place as we know from openness of cell that's actually also a good idea openness is L. use it's all in memory located I think it's probably still doing that but we threw it out and the rest of the rest of list using the system out of solely wrestles Alice not doing the exploits mitigation mitigation anymore that all the necessary used to actually at that surprised me a lot that a few months after we read that this and of Mrs. element and removed it from from the lever for that phone all this custom all located the other way so so OK that's a design decision for performance that makes sense there but we don't want that you want or pardon mount adults randomization and a new self-defeating detection and so on so the privilege separation are really processes that communicate which is of the parent forks them the beginning and then they just run there's no responding of something that you can configure the the number of so processes and then each server process handled uh the the connection was an increment I also best no spreading involved or something like that and the server processes for example don't have right except for the locked file they send a message to the long process basically the nice side-effect as so that you can have multiple sober process and the messages to the single longer get serialized because of the messaging and the performance of those really good so and we we can open lot fights was right privileges of our competent will to the other web sources but the the server processes don't have to touch and there are some other things so the over process for example they also run with an unprivileged user of the they began to anything harmful and if we ever need another thing we might add another privileged process and we lady for example we have another process for all the hours a private keys and and and optimism TPD again at that 2 http yet but I will at some point so that when we did the TLS
so the aggressors L added annually API on top of that it in the beginning it was called live re ethos of what this wasn't quite confusing because live we as a cell phones likely resistor but and but this is actually a part of the resource of so it's called the TLS simply and it's basically an API on top of it but it's so easy to use you should really have a look at it you can ride Keeler clients or servers in just a few lines and it does everything right so Joel thing as doing the the major work there and I'm doing it from a reference implementation point of view so i in ACT PAT we decided instead of using lipases out directly the old API that uniform open of that we we use here so this also helps to shrink the size of HTTP and by default it only don't you have 1 2 for some months now and only it's my cyphers and so on so not for example wasn't an issue for GTP Fast
CGI assess that was contributed by a story and also I have another German I asked him can you give me a quote for the presentation why did you implement Fast CGI and he said I implemented CGI that was the CGI record that we had before they implement loci to be united because you didn't stop whining on ICT that engine kind execute BGP of and Fast CGI in HTTP balk at what has often me if I can help you with so a little bit back when we remove Apache there was no run to run the BGP looking glass glass anymore because it is a classic CGI and and genetics is not supporting the classic CGI interface which is the right way to do so we needed a Fast CGI support in in the BGP looking-glass or a Fast CGI wrapper so overwhelming showed up and wrote this slow CGI which is basically a little so that helps you to run traditional CGI eyes and then talks was Fast CGI to the uh web server and he uses code later recovered new implementation of officers CGI protocol without depending on the official libraries and all this bloat the use this tool to ride the Fast CGI silver code for HTTP on which works we we we well and we do have a direct streaming there's no intermediate buffering to a file the configuration that also an example that I hope you can read at the next slide I will give you an example of the basic web server consideration you open a text file and put that in the text file http the . com and then it's working a case that's all you need to actually have been thinking about making the listen on port 80 as a default as right so you can run it was an empty file or something like that but that's a minimum required so yeah this it is the sort of yes if if we don't do Reggae Reggae exit but we at the moment we do support that the an F and match growing Rhodes so you can do like shall show why can't physically but what people already reflected you start . example . com and as virtual hosts from the world the most of the time he was the home and that the here I think on and so on and so it is really the way it is here that you want and since then even added that like name-based aliases and all that that helps to reduce its 1st on my case so that's a bit more advanced for example you can include an external mind types file if you don't do it it provides a list of the most common types like HTML JPA jobless but just otherwise you can just use the existing lineup types fights competent to the eparchy slash and genetics of form the for that we even for the MIME types we even posse semicolons at the end of the line because if you see we don't need chemical the and from what the grammar is and using the same part of the word that PS CP-odd down the path of white comfort Pierre for GPD lady we we use it in many places and openly is the right now that's all unified configuration actually without breaking LS them the region and the way so not without using an external library of something that there an obvious thing we just reuse as part of white coat that originated from and PS father and then we use it in all the other new demon the GPD anti pd even relate the all of them and so all you can use macros like and Pierre of you don't have to write semicolons at the end of the line and it's very feminine on some advanced configuration very bright and with it so all you can listen more purports you can also have additional sober names for for name-based aliases building is enabled by the default but control and in all locations is the matching of all using and match at the moment is a set of of very not going to direct regular expressions so there few options they're all documented among page as usual and obviously hasn't a montage of really good shape and so that you can understand what it's doing and it's not like long and you don't have to pick up from the webpage also just you monetary pt . com the blocking roads so support 4 redirect you can redirect and so on faster during a few other options that works well in combination with PHP as program of course but also with was many other frameworks future work that is the 1 that's very new and actually not even all of the of nifty developers about because it hadn't been released yet and I CEO is working together with me and a few other developers the most of the work was done not done by the name of the snake and and you will figure out when it's really saw somebody had no nifty implement something CEO is designing and we we're working on a new framework to improve privilege separation into further brought privileges but it's the line in a way that it's practical and some practical approach it's easy to use so basically the kernel then that's the the interfaces to to a substance that also the pose X and the environment that you need in the individual process and it works really well with privilege separation for example http http the longer process doesn't have to open any network sockets so we can we have a class basically that we can drop it so much easier and better line than Lexus trace for example on T other things and other systems but it's not trying to solve every possible problems it's trying to be a practicable practical approach so stay tuned actually will be really nice and we will use it everywhere actually so more features are in preparation that the SNR is supported by a promise that before but that the program rewrites well not where is a regular expressions but we found a very nice way and that is currently being investigated so that we can do read riots and advanced matching um but was matching language that that I can understand where can be the source code and know what's going on missing I asked my glucose what do you think about regulated professions and that all people are asking me all the time to write a book about regular but why do we have to to write a book about it that at this time when it's so complicated in the 1st place and I don't want to use something in it the PDG for the pattern matching where you have to read books and books to get across so we found something else and I hope that I can release more information about soon but actually I just started looking at yesterday so you have this time in I think attainment in English I think in in in japanese that also has a nice meaning so and so the game I will limit the privileges of each process so you can decide that the server process is not able to and I'll do set change the the system time more for the love of God and using the example is good that it doesn't have told many network sockets and so on so this is once again very easy to use and that will further improve the security of it should give PT but it's not specifically for it should ppt mostly everything in baseball yeah so often this affects the 1st 5 7 was released in May by the use of the product the projects and have a look at the funding come paying for this year and by of beer actually so thank you