VirtualHostMonster+nginx: hosting 250+ Plone sites from 4 installations in a single hostname
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 39 | |
Author | ||
Contributors | ||
License | CC Attribution 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
Identifiers | 10.5446/47846 (DOI) | |
Publisher | ||
Release Date | ||
Language |
Content Metadata
Subject Area | ||
Genre | ||
Abstract |
|
Plone Conference 201324 / 39
1
9
10
12
16
17
23
25
29
31
32
37
00:00
Plane (geometry)Position operatorDisk read-and-write headNeuroinformatikSingle-precision floating-point formatPhysical systemApplication service providerWebsiteVirtualizationEmailPresentation of a groupMonster groupArithmetic meanDemosceneXMLUMLLecture/Conference
01:40
Plane (geometry)Point (geometry)Multiplication signWebsitePlanningNatural languageProduct (business)Structural loadWindowConfidence intervalObservational studyDifferent (Kate Ryan album)Process (computing)Type theoryInternetworkingControl flowFile archiverWeb pageDirect numerical simulationProjective planeArc (geometry)Set (mathematics)Server (computing)Metropolitan area networkUniverse (mathematics)Parameter (computer programming)Computer scienceGreatest elementConfiguration spaceLibrary (computing)MappingContent (media)Single-precision floating-point formatArithmetic meanFigurate numberTorusEndliche ModelltheorieSystem administratorProxy serverDirection (geometry)Presentation of a groupFile Transfer Protocol2 (number)Software testingReverse engineeringMultilaterationUniform resource locatorCodeDomain nameNeuroinformatikService (economics)Mixed realityAddress spaceLastteilungLecture/Conference
09:36
Plane (geometry)Computer fileProxy serverWebsiteSoftware maintenanceAddress spaceServer (computing)VirtualizationUniform resource locatorFront and back endsConfiguration spaceData storage deviceReverse engineeringRule of inferenceProduct (business)SoftwareMonster groupWeb pageMereologyFlow separationMetropolitan area networkEmailClient (computing)WordPoint (geometry)Operator (mathematics)Virtual machineProcess (computing)ChainObject (grammar)PlanningDemoscenePhysical lawGroup actionThomas BayesDistanceDisk read-and-write headContent (media)Lecture/Conference
13:31
Plane (geometry)Line (geometry)Generic programmingMereologyPoint (geometry)Product (business)Configuration spaceServer (computing)Software testingData storage deviceSelectivity (electronic)BitSingle-precision floating-point formatWebsiteUniverse (mathematics)Virtual machineHuman migrationContent (media)Event horizonComplex (psychology)Address spaceElectronic mailing listReverse engineeringUniform resource locatorClient (computing)Proxy serverVirtualizationFlow separationWeb pageScripting languageNumberInternetworkingHoaxComputer fileWeb serviceWeb browserTouch typingField (computer science)MultilaterationFamilyEmailChainCASE <Informatik>Type theoryRevision controlOrder (biology)Standard deviationMultiplication signSimilarity (geometry)Forcing (mathematics)Theory of relativityProjective planeService (economics)Row (database)Patch (Unix)Coefficient of determinationHypermediaBoss CorporationLecture/Conference
21:01
Plane (geometry)Multiplication signLink (knot theory)Slide ruleLecture/Conference
22:36
Plane (geometry)1 (number)Lecture/Conference
Transcript: English(auto-generated)
00:11
Just like a teacher told me, the Linux is not a presentation operating system. Sorry, I expected that I had a computer here.
01:07
Hello, my name is Everton Foscarini. I am presenting here virtual monster plus nginx. I have an update of the title, hosting more than 250 open sites.
01:21
And now from five different blowing backends, more than 500 Apache and 200 ASP sites in a single host name. Who am I? I am Cesar Jimin in our university, URGS. I work mainly on Linux and email, the email server, the
01:51
And the most work is in virtualization. What is URGS? URGS is the biggest federal university in the south of Brazil, established in 1934, and has internet presence
02:04
since 1990. This is the oldest archive in the archive.org from 1997. And then we have a major re-stilization in 2004.
02:21
That lasted until 2011. And this, the new 2011 to plenty site that is available to now. Why all these sites? Because the university has a lot of history in the internet. We have hundreds of sub-sites, the department sites.
02:44
The main site is a sub-site of the main domain. We have the IT department, the e-learning, the computer science, the medicine, all the main departments and laboratories and libraries, all have sub-sites.
03:03
And many URLs have more than 20 years. So legacy matters here. We had some challenges in the hosting service in 2010 because we have silos. Disconnected hosts, the main site, www, was Windows and
03:25
IIS, the second, Linux with Apache, and the new, in 2009, we started using Plon3. And have a lot of others, different host names, www2,
03:43
www3, www4. I think that 11 is the last one that we have. And mixed types of contents, mixed types of servers, and a lot of different administrators. And all the websites want to have a www URL because it's
04:09
built here, cooler to have a www, others not, a www23 as you will get to put in documents, in material.
04:23
And the IIS had hundreds of rejects just redirecting to the www6, www2, the paginess. And when a site was migrated to another, to Plon3, we have one more redirect to configure.
04:41
Other problem that we had was that Plon3 had multiple host names. The first configuration, the Nginx, has had a mapping of the host names, from the host names to the sub-site.
05:01
So the same site had three addresses, CPD, the IT department, www CPD, and pages, paginess, slash CPD. And we have also the redirect from www URL slash CPD to the paginess slash CPD.
05:21
So we had at least 40 sites with this configuration, the same content, three different URLs to Google to index, and Varnish had to cache for three times the same object. Other challenge, we had to start using Plon4.
05:41
We have some problems of performance with Plon3. And the problem to upgrade was that we have some spurious products that didn't behave correctly and could not upgrade the site to Plon4. So what we should do, create a new host, paginess 4.
06:06
We also had to upgrade the Linux server. It was a Debian 4, very old, full of broken sites and holes, Debian 4, it wasn't supported anymore.
06:22
But we could not just upgrade because we couldn't break sites. We had PHP 4 and PHP 5, and the users didn't upgrade their code to use PHP 4. And we had more than 300 sites to test in a upgrade.
06:45
So should we create another new host? And in the end of 2011, we have to change the site or migrate to Plon4. But we still had more than 300 sites using the Windows
07:05
server that has the www address. So we could have three possible solutions. Or we create a new site, new URL for Plon4 and create just one redirect. We can steal the address from the Windows server and
07:23
rename the Windows server and have 300 redirects to configure. Or we could create a reverse proxy, the only same solution, and clean up all this mess. So that was what we have made.
07:41
I will share the presentation later. This is the current status. We'll talk about this later. I made a first try in 2010 just because I wanted to have a reverse proxy in front of the Windows server just to take it out of the front of the page so I can have some
08:05
redirects, some different content. But I made a single host. I had a single point of failure. Apache had problems. It was just Apache without modulus, without specapp, had some problems with the medium load.
08:23
It wasn't in the SATs, the visible R. The time that we had more than 40,000 visitors. It was a normal day. And the users used the FTP connecting to www.
08:45
So I have to have FTP portrait reactions just to don't upset users. So I keep for one week and must have to roll back.
09:02
A second try was made in 2011 because we have to migrate the site. So this time I made it right using GenX and load balancing. Load balancing since the beginning to have mainly fault tolerance, had a good performance, and the FTP users
09:25
must use the correct DNS server. They spent six months after changing the DNS. The users are still calling to know where was the server. So they were obliged to use the correct server.
09:48
So the main goals of the reverse proxy for the hosting part, not for the main website, was that any hosted site must have the www slash site address.
10:04
So the sites must be either in IIS, Apache 0.3, 0.4, WordPress, any backend should be supported. And the website can migrate between hosting servers
10:24
because they don't have to make a redirect. The content migration, I don't care. The users want to have the address. So the address are theirs. So we can migrate from one backend to another without problems.
10:42
This was the first production setup with the reverse proxy and the IIS, Apache, and 0.4. I had three servers, not just one reverse proxy, one just to make an example here.
11:03
And we had 0.4 supersized infrastructure because of the problems of 0.3 installation. We were very cautious, created two varnish servers, separated the 0.4 servers had just varnish to have
11:25
fault tolerance, and two servers with clients, each one with two clients. And the server, it was in a better machine with storage.
11:44
Trying to, the ZODB wasn't fault tolerant, but it was a drawback because the ZODB replication, it wasn't free software in 2011.
12:01
So the configuration in GenX, it was made into parts. I created a location for the backends here in the down. Each backend has its location at the beginning. And to define to each site in which location it should
12:24
find, meet the request, I have the at, plony, at the end of the try files. So I have a maintenance page. If I want to remove the plony from the website, I can just
12:42
create this HTML. And the GenX will provide this page just saying that the plony is in maintenance. I had the setup to have a simple configuration for the backends. But the talk was about virtual host monster.
13:02
VHM is used to generate HTML according to the rules in the URL. What means this? I can have, this is made in varnish. I can say if the site is public, use the www.orgs.
13:24
If the site is to management, use the plon.orgs. And the content is the same, but it is generated according to the URL. This is a request directly to the client where I can show
13:46
the virtual host base. And the fake test URL. So the generated HTML has the fake test URL as the host name, part one. Totally fake, but this shows that plony generates the
14:03
content according to the configuration in the VHM, not in the requested host name from the web browser. So going back to the hosting. Now, the varnish was in the middle of the field, but I
14:27
now move it inside the Plony machines. So we can have a granular configuration. The Plony works, the main website has a more aggressive
14:41
policy for cache, and the others have different. So the main website has the content managed by the public relations, mainly news and events. Uses memcache and the web service. The web service takes from the internet, from the
15:04
institutional database, some configurations, some list of teachers, professors, list of courses, and this data is stored in memcache to alleviate the server.
15:23
And we have a talk about it today, about the site. Plony pos is a kind of internet for several graduate schools, uses the same configuration. We have the Plony hosting, hospitalizing. Generic hosting is where we have the more than 250 sites.
15:45
We have a restricted product collection, selection just to don't have problems with products. We have a talk about this tomorrow. And we had some migration from Plon3, PlonPaginas, and
16:00
Plaginas2, because they were migrated in different types, and have different versions. And we have the new Apache server, whose migration was made slowly, side by side. And the old IIS, and now a WordPress.
16:21
How do we configure Nginx to deal with multiple back ends? I have multiple locations. I have more than 10 locations connecting to back ends, just like before. So how do I say to a website, this is an example, lab solos, which location use?
16:42
It should use the location colossus, is the Linux Apache. The other site, Progesp, is the human resources, uses the migration, PlonPaginas, the migration. And here I make the first part of the VHM configuration, where
17:02
I say here, after VH is the final URL. I'll talk about this later. But this kind of configuration for each one of the 800 sites is a bit complex.
17:21
So we created a kind of reverse proxy config file, where I can say for each URL, in which back end it is hosted. And we have a script that generates the Nginx configurations, and should just reload the Nginx
17:42
configurations done. A new site, so it's just two lines, one line. The final touch in the hosting, Ploni, we had multiple datafests. So we can segment the department, the sites by
18:01
type, departments, laboratories, libraries. Look at this as a good idea in the beginning. So we can have separate partitions, can enforce quota, have smaller backups, less DB packs, because some sites are less dynamic. And maybe it was better before blob storage, because
18:21
blob storage separated all the big files from datafests. And we had a final drawback. The ZODB mount path must be in the URL. So we had to redirect again. But VHM comes to save the day.
18:43
And just like I said, the VH configuration says that this is the URL, CPD, doesn't matter in which mount point, or even if this name of the site was other, doesn't matter, the VHM can create the correct HTML.
19:07
So we have here the hospitalizing, located in the mount point to Unidad with the VH CPD, degenerated HTML has just the CPD sub-site.
19:28
Conclusions. We have success enforcing a single public host name. And we created a GenX redirect server just to handle old virtual hosts, so this site has 40s, 50 virtual hosts just
19:43
redirecting. And we made a policy. Every site should have the www.wirs.br address. And these are the numbers of the sites still now, always growing. The reverse proxy configuration file make it
20:01
manageable, create all that redirects for each site to be painful. And VHM gives deploying a great flexibility. We can rename a site. We have already made it, a department changed his name.
20:21
We just had to create a new line and say, now you should use the other name. This VHM can hide mount points. I don't remember. And with VHM, we can create a different URL, in this case,
20:42
plowing.wirs.br, that could have been accessible just inside from the university to don't enable the authenticated pages to be available outside. So now we can migrate backends, maintain the same
21:02
URL, and I think that I don't have time for questions. You have my contact here if you want to make a question. I will publish a link for the slides here later. And thank you.