Welcome to the second talk today, a technical topic I would say. It's about scaling up Plone or making Plone fast even in smaller setups. My name is Jens Klein, I'm located in Innsbruck, in Tyrol, in the Alps, Austria, developer

since Plone.1.0, also did core development this time and today.

Also I love open source software, so it's the reason maybe why Plone and the community behind is a really good example for open source. My company is Klein and partner KG and we are member and founder of the Blue Dynamics Alliance which does Plone also that long. I have here a little advertisement, in short we do a sprint in January in the Alps if

you like skiing and sprinting and good beer and a vibrant city in the middle of the Alps just join us, registration is open. But back to the problem, Plone is by default not that fast like any application

server, it's not different from Drupal or other system. Even maybe pyramid is faster but it has not this lot of functionality. So Plone scales great, you can just add machines, that's what system administrators

love but sometimes adding machines is not enough. And so we have other bottlenecks left. One bottleneck is the database, always, so loading stuff, data, pickles from the ZODB

needs to be optimised, then the rendering of the site itself is the bottleneck, page templates are usually slow, then searching the information you want to display is slow, a portal catalogue is relatively fast but you need to use it and so on.

And then you have maybe to talk to a third party service, that's a problem too. So you have to know your knobs and handles and that's what this talk is about, I want to give an overview of the knobs and handles, you can use and turn and press and where

you can hook in to make it even faster. It's in a submarine there, a Russian one, old one.

So I have picked out some customers with different set-ups to go through the talk. One customer, they are in central service for cultural institutions, in short NUCCU, Nieder Usterreich Kulturwirtschaft in German, so they have over 30 plan sites running mostly

for three or for two sites, they have a cultural, for cultural services, so the budget is not that high but enough to make it running good. They have own hardware and own data centre, so we are running it there with virtual machines

on this two bigger or three bigger servers and do high availability with pacemaker cluster resource manager, the block device application.

I apologise it's in German, but I have this picture not in English. The classical set-up is usually that you have, you can see it here, web server, reverse

proxy and caching proxy, a load balancer, a clone instance on several machines and So we are in this set-up, we did it a way that if one machine is for some reason gone, then the other machine can take over the whole load, the database, switches over

to the above machine or the other stuff switches down, but the clone instances then are only on one machine, one virtual machine, so you may not handle peak load with it but the usual load is fine. As a customer, using clone for the portals, product portals, B2B shops, externets,

customer communication, they have different set-ups, they have a hosting contracting partner doing the hosting management, but the services we put on it like clone, we

have to manage ourselves most of the time. Also, the storage is on a network attached storage, so we have the input I.O.

problems with this thing, because it's not that fast. Another one, smaller one, is a student representative body from the university of Graz, and they are having one site with several lineage subsides.

So at the beginning of the semester, the students are all going on the site and will have a peak load at the beginning and afterwards it's more flat. So it was really bad hardware, they have a plan for five years for the server, so it can't update it, and it's always virtual machines and completely overloaded

hosts, and we have to make it fast anyway on this stuff too. So, really different set-ups, and if you then look into clone and all the stuff behind, it looks like that. A lot of bricks and you can put them together in different ways, but how to

make it fast. A bit to the basics, because it's all the places you can hook in. It's a database itself that can be different databases, then you have a kind of client connection pool into the SOAP server.

Then you have the clone itself, so the application threads running, maybe, or you usually have more of them. Then you have the load balancer, to have the load on different machines. And finally, publishing front-end, like the Caching Proxy Varnish web server,

and finally the web browser, which is also part of this chain. So, usually we use a Zoo server, or write storage, and this write storage is on Postgres.

And always using block storage on NFS or NAS. So, the client connection is usually the different Zoo clients, or the storage adapters, which are in fact also Zoo clients, like the write storage adapter. The clones are the clones, as you know them,

and a load balancer we have in the past used Pound and in the last two years I think we switched to HAProxy. Caching Proxy Varnish, Squid is past, if you use Squid, better use Varnish, really, do yourself a favour.

Same with web server, I think Nginx is also the first class citizen and Apache is kind of past if it comes to the scenarios we have here. For web applications, maybe Apache has in other areas its field,

but I think Nginx is fast, better suited. And finally we have the web browsers, and you know, yeah, the browsers. And then all the clone instances are talking to third-party services like SAP Business Connector or whatever you have there in the world.

And this has to be fast too. And also we're using Memcached. Memcached is primary to cache stuff and is used by both the client connection pool and the clone.

And so you may need more than one Memcached instance to make it good. So you have to consider also how many CPU cores do I have, the ratio between used RAM and disk cache available, the IO on networks, the IO to the disk or to the NFS or whatever it is.

And so, yeah, a lot of stuff. And to get all this together, first thing you have to do is to monitor all you can do, all you can monitor on your servers. So Munin is our tool of choice there and all the others around.

With Munin you get nice graphs and you can see over time how the load, how the memory usage is, how the ratio between memory and disk caches, all the stuff, CPU load, peak loads, IO weight of the CPU. You get all the base parameters.

Then there's FIO, it's a little tiny Linux tool you can measure disk IO with and it's really good to get an idea how fast your disk IO is. Then you really want to know how Linux, we are using only Linux, maybe BSD is different,

I don't know, but Linux how it manages disk and RAM and it's all the caching stuff. And for sure you have to know on what hardware you are running and how you which your machines in the cloud or whatever these virtual machines are if you have some, but you have to know

what they are really doing, otherwise it can get really slow. So first we have now the database. At NURQ we are using a ZIO server on a replicated blob storage with DRBB

and also there are some other services replicated and the blob storage is also on the same machine and is also replicated. So if one machine fails, it fails over to the other and the whole switch takes under one second

so it's really fast. So if you unplug your machine it takes one second and everything is green again. ZOOM tool, we are using a storage in the beginning we had a really large database we did all the product imports there

so it was about 300 million objects in the database and the storage just managed fine and the blobs and all the data are on a NAS on a network attached storage over NFS. And in Graz we have only one machine

and we are using anyway real storage on there because it behaves a bit better than the ZO server for this low budget server for this overloaded CPU cores and so on.

And first things first maybe most of you know it but this is important never store blobs in the ZODB. So if you use Plone 4.3 out of the box and you use news items with images the images are in the ZODB change it, it's possible there are Google for it

there are menus how to do it use blobs storage and also for your add-ons check it, what they are doing with blobs what are the images really if you put blobs in the ZODB in any way even with real storage it's worse but don't do it.

So if you use a classical setup ZO server and blobs storage then you need fast IO to the hard disk and a ratio a good ratio to have your database cached in the disk cache

from the system the kernel cache. That's most important although there are not many settings on ZO server I think to really reduce some overhead it's really fast as it is and I think that's fine.

The blobs storage can be configured in different ways so it can go through the ZOOB connection or you can have shared blobs what we are doing is using NFS and shared blobs so that we mount the blobs storage on the clone nodes later on

and you can also mount it on the web server itself, maybe read only you don't want to write that and deliver the blobs directly their mechanisms with this collective accent file which sets the HTTP, Excel or the accent file header depending on the web server

and then the web server really delivers directly from the file system so blobs and that speeds it up a bit and reduces your network traffic inside because otherwise the blobs would be fetched over your network from the database through the clone to the other machine where the varnish is on

and so on, get all the chain down and if you can just tell Nginx get it directly from NFS you don't have it three times the traffic if you use the storage it's slightly different

it's the same but much more important is never stop blobs in the SQL DB maybe with Oracle I never tested it, it's possible I heard about it I don't know do not use MySQL if you can avoid it it works

read a bit faster and if you have concurrent writes you can have problems with it and if your database grows and you don't use the history free mode and you want to packet it the consumed size on the file stays the same

also ensure SANE and Disk and Storage IO check it use FIO this tool and I have Ram left, I talked about it so if you compare both Zio and the Rails storage adapters there is one big advantage on the

Rails storage side the Zio server has a connection to the Zio client and always tells the Zio client if some object is invalid or new transactions are available the Rails storage adapter polls it and the polling

times the delay between saying it's outdated can be configured so if you have high read size or instances configured for reading you can and you can live that delay between having the most recent data is 60 seconds

after pressing save you can have on peak load times where you have really lots of requests on your server you can have this delay available to not overload the database back end it helps a lot and on the Rails storage side

you have not this disk cache which in fact is also a memory cache because the system caches the disk part in memory and so we never using the disk cache and having only the memory cache by per thread and on Rails storage side you can have

it in memcached memcached server can be shared all the pickles loaded can be shared between all blown instances running on the same machine and so it's yeah it's a big advantage it reduces again load on the database

and the RAM cache of unpickled objects per connection the usual RAM cache where it says cache 30,000 objects can be then reduced so you can have more CO clients running doing parallel work on the same machine if you have enough CPU cores

and not memory is the limitation so short to the customers we have here 30 sides and it's a very classical configuration maybe most kind of default configuration

except that we have the DRDB DRBD and we have two threads usually sometimes four and 30 sides and 30 and 60 blown instances mirrored

on the different machines sometimes we have history free Rails storage history free means usually you have an undo in blown and the transactions are just appended in Rails storage or in SQL it is an insert but if you have search to history free modes

it's an update on the current row so the database grows not that much then but you have no undo in clone if you don't want the undo which does not work very well then it's okay

and it speeds up the things a bit again there are two virtual machines for the clones on each virtual machine 16 instances with two or four threads

and we have also worker instance asynchronous tasks like import database syncing with the product database very special customer specific stuff but we have also memcache installed

and share the connection cache and that speeds it up a lot without memcache it's way slower and if you have that many objects in a Rails storage I don't know what the limit is but the usually packing mechanism of Rails storage consumes lots of RAM

memory and then you have the problem you have a problem with packing because on your machine all memory is consumed and at some point it says I don't want anymore so if you really have this problem look at this Rails storage and let's go packaging on PyPy which solves this problem

also checks for 300 million objects normal packing mechanism estimated 300 days of packing that's a lot and now we are down to one day or eight hours or something like that I think it's a new one takes a while anyway but it's okay

it runs on the you can run it on any machine so it doesn't have to run on the same machine as the instances are running and our small setup which really has also high traffic on it and has to render lots of sites we have six instances

with one thread and a RAM cache of 30,000 objects and interesting the interesting thing is to reduce on this bad hardware it's better to reduce the number of instances and make them deliver fast

and increase the number of objects you have really in your memory cache unpickled between the six threads the unpickled parts in memcache again and you have really to try them how many objects do I have in database how many are loaded all the time

looks at the catalog all the catalog fits into this 30,000 objects and so on and so on and we then use it as a poll interval at 120 seconds 120 seconds means that if a request comes in on a machine then it takes up to

120 seconds it takes it from the memcache and if this time was passed then it asks the database is there something new that helps also a lot it's all about database and it's really on sync point where you can really speed up Plone on this side

yeah, but Plone this was all database it's all valid for any Zope it's also valid I think for pyramid applications running on geoDB I think substance D is one of them but it's all not Plone specific in what I know

but now what can we do about Plone itself and that's the problem we have a request, we have a response and they're not Ploned us so what is it classical configuration issues we can turn off debug mode

we have a security, logging critical deprecation warnings we have to configure on a default Plone 4.3, Plone app caching because it's not installed you have to install it and switch it on even if you don't use Varnish if you have a really small site it's a good idea because it speeds up syncs a lot

you can have then there's multiple Plone instances and you can configure even Plone to use memcache instead of ramcache and you may want to check JARN check interval which gives you a value you can you can

put into the configuration of ZOPE what's the check interval of the global interpreter lock is I think so this is all documented on Plone.org how to do it so it's nothing new just keep it in mind there's a good section on Plone.org

in the docs the default Plone is almost not a problem it just really works good also the caching parameters are working fine but if it comes to add-ons and I think all of you use add-ons in Plone you have to take care about them if you write custom code

noplone.memoize and use it never calculate or search things twice I often saw in code that you have a thing in a template like tell condition and ask a few dot results and then next is tell repeat few dot results

and so it fetches the results twice and if you can avoid this do a tell define before or just cache it with memoize if it's the results that you can use over several requests unmodified you can build up the same cache key

then use Plone.memoize to cache the stuff and don't calculate things twice also in Plone you can take care that not only results directly are cached but also the prepared data so you get something from the catalog and need to filter it again for some reason

or need to get some data from another place or you talk to a third party system like a SAP business connector or something strange you have no idea what it does this third party system but if you get data out of it, cache it just that makes things fast and those are the usual bottlenecks

if I go to third party add-ons to make them faster I find all this stuff in several places not cached yeah, I say again never store blobs in the UDB use the catalog and don't over use muta data so what I always see is that I put everything

to muta data and then if you search the catalog and wake up the muta data it's not much different than waking up the real object and because there's so much muta data in the catalog that it doesn't really make a difference it's crazy then a big problem in Plone

write conflicts and write conflicts are 90% of the time write conflict means that one transaction wants to write the same object, ZOID ZOID thing on the database level then another transaction wants to do so if you have two

writes and they don't collide, it doesn't matter but usually this happens in the catalog and the catalog consists out of B-trees B-trees are consisting out of this buckets where one bucket is one thing, one ZOID in the database if you have to think of portal type

and the portal type index and they have the same portal type saved twice and they want to write the same bucket it gets a wide conflict and write conflicts are really a big problem the first thing you can do about it is to reduce the time of the transaction if you can do it and you're fine with it and it's good for your setup

try to make it really fast so that the transaction is really fast but sometimes it doesn't help because you really need to prepare data and check things and so on and then it comes to the point where you say okay let's use something different than the catalog it's the only way, in my opinion

to get around it and we have with Solr very good experiences at this point and I can really recommend using Solr my big vision is that in Plone 6 we have no catalog anymore I'm just using Solr that would save us so many problems

yeah next problem you have lots of objects small Plone content it's easy to make it fast, but what about if you have hundreds of thousands of objects and I think

it's you can really store lots of objects it's no problem you can load them but again, if it comes to the catalog it's a bottleneck also, what can I do about it is collective Solr and we have the scenarios

and yeah if it's not that many objects you can put the catalog on an old mount point and give it a real large memory cache 200,000 objects while your usual other Plone, the root mount point has only 30,000 objects

and that would work wire, but at some point this doesn't work anymore and then you will have to use Solr now, in Plone 5 we have dexterity and now we're used to work with archetypes last years and

always there was a mantra saying don't wake up objects don't wake up objects wrong with dexterity? if you have a folder and iterate over the items and you have things like simple images of files in it

do it, iterate over it wake them up it's the same as waking up the metadata of the object because in dexterity the objects and the space used in the database is very small it works out fine, you can really wake them up it's an interesting thing, we really tried to make it with catalog or with direct

iteration and it works you can iterate over it and with dexterity if it's not something insane then it's really a small object you wake up and it's fine, you can wake it up but if you have archetype stuff really don't do it it just blows up takes lots of time

and if you have some custom dexterity type and it's all stored as attributes directly on the thing it may be also better to not wake it up, so it depends a bit on but if it's a classical standard clone up content types it's usually okay

yeah this adding of metadata to the catalog is really a problem I saw it and some projects we got from other companies and then I looked at everything it was put into metadata there was a bunch of metadata added

and there's also a thing where the catalog really blows up and gets big and also the loading of the metadata gets slow again and then if you can avoid adding metadata don't add it if you use archetypes you may want to do it and if your dexterity is possibly cheaper

don't wake up the objects and also the yeah, the third party services, the next one need to be cached, that's what I said again

and you need to also think about how failsafe are the services all the problems that you can really stop an instance or thread an instance, you block it because the service doesn't answer and it waits until I don't know, kind of 300 seconds it should wait too long and then the user tries again

and clicks again on it and then it blocks the next thread and at some point no thread is available and then it blows up you just have no threads left and your site says 500 service is available so if you have serious performance I would say

first thing you can do is measure so don't try to configure something around and get real first data out of it and then you have some tools I wrote I will put the slides on slide share or something like that and also the video, but

if you then change something, don't change more than one parameter I saw that people change four or five things and you have no idea what happens because change one thing, even experienced IT stuff I watched them really trying to change five things at once

and so I think I have to tell you to do it, not change one. Then next thing is on the front-end side after the application server, clone app caching I think there is a lot to say about clone app caching but if you have a plain clone false result

you just can use it, enable it and it works fine if you have custom add-ons you really need to want to configure it and it's documented well how to do it also on docs-blown-org and it's really the part you want to configure if you have own add-ons write your custom code

and with Varnish together with clone app caching are really our superheros here to make clone fast because it really makes it deliver the pages directly from Varnish and you can get really really high traffic on a clone site with it and have cache hit rates over 95

up to 98% if you do it right so if you monitor your Varnish with Munin you see the cache hit rate and on peak time it gets all faster so we have the problem on a low traffic time in the evening at 11pm for example

the site is slow and then at 11am everybody clicks on the site it's really fast so Varnish is great and one thing is this micro-caching what we did so we added caching rule that just caches stuff for one minute or for two minutes or five minutes depending on the site and this really

helps also to speed up the things on this high peak load to really have recent content maybe takes up to five minutes that it's on line but you have good caching experience and it really buffers the peak load we use

it on some turbulent site and it speeds up things you see it also on the Munin graph it's nice to watch how the peak goes up to 90% or something like that then we have load balancer

and what to say pound is old but you don't use it anymore on recent setups and JProxy is a new thing it's more complex to configure but you can do more things with it important point is session stickiness so you can use it to direct one user or one visitor or one language group to

a group of clone instances and that way this group of clone instances can better cache the data from the database because the cache really is filled on this part of the cluster with data used by this group of visitors so you can really cluster your groups

of visitors by language or by by customer type on one external net we have or by you can cluster by different editors or non-editors and so on can configure the instances different then we have the web server

it's just working good the only thing about the web server is that you set the proxy underscore stuff variables right this is what usually and all of this documentation same and you have the

possibility to use this collective extend file to directly deliver blobs and save some traffic on your network yeah that's oh yeah the cat image I think every talk needs a cat image, this is things called bobcat right looks in Germany

we have some, some kilometers south of us in the national park so I took that one yeah this is a chat on the blown documentation performance and tuning, I think it's very good there's lots of information and you can read further into that

and yeah, have you any questions? have we see a microphone or yeah, okay

probably a lot of us have begun deploying on systems that have SSDs for their drives in the past year or so and I mean when I've done that I've changed several things at once so I haven't really been able to figure out the differences and I wonder if you Jens or anybody else is developing a good

feel for the way you switching to an SSD changes the way you might balance your memory for efficiency SSD is the next thing that comes up but on our servers we don't have SSD we're almost on the network attached sort of stuff but I think with SSD it would need new measures

how it works this guy gets faster and things are different, right? I mean, I ended up running more instances with less cache than I ran before and it seemed right but since it was a new install and I changed several things at the same time I couldn't measure the differences

I was going to ask one thing we've noticed with dexterity is we're expecting it to be a lot faster than archetypes and in certain situations it can be about four times slower when you're trying to look up an attribute on dexterity because it has to work its way all the way through the schema

and compute where that attribute might be Do you know if there's any work on speeding that up at all? Yeah Okay Dexterity is a special topic maybe if anybody is interested in dexterity and what's wrong with dexterity I had some last two weeks

I really dived into it I also know because it's slow because if you have a factory and it has to look up the factory and all the stuff on a behavior then it really takes its time to get an attribute if you have direct attribute access if you have a main schema or no factory with the dexterity schema

you have direct access to the attribute and it's fast if you want to have it really fast you don't have to use this factory but it's also a disadvantage I think it's fast if you're looking for something that's in the schema but if the attribute is not in the schema like if it's a base attribute like some of the internal zope double under ones

then it can be really slow Yeah, that's a thing we need to solve but there are other things to solve in Plone.x too I was just wondering so you talked about how you measure the performance of your server you know you can use Munir to measure your disk IO

and all that sort of thing do you have a good way of measuring if you like the output of your site how many page hits they're serving or what sort of network bandwidth your visitors are getting as opposed to the output you're getting from your disks if you see what I mean the real sites we can solve

I didn't get the question right I think So it's easy to measure how much IO you're getting from your disks or whatever but in terms of how many pages per minute you're actually managing to serve or how much network bandwidth you're getting out to the users rather than from your NAS or whatever Yeah, it's difficult to measure we had a tool what was the name

it was a really crappy windows tool but it was good to do the job and I really need to look up the name and then there are several tools out where you can have this click passes through the site and also vary around and log in users and such things so if you really want to know

the behavior of your site on peak load you need to have this kind of measurement tools I think there are also cloud services around to measure this but you can't get this into Munin I think external measures and then you have to try how how does it work so

yeah, it's a different topic to get this data sorry? you can request per second and also Varnish tells you the Varnish Munin log tells you the request per second yes

device per second yes right all the Munin plugins are showing this information but if you really want to know how many requests in a certain scenario I can get out of my systems you need an external tool to penetrate it

yeah in some slides you told us about reducing the time of transaction my question is how to do that yeah it's primary to reduce the time on transaction on save of a classical

content type it's difficult but we have also scenarios where we have forms and then save the data ourselves and there you can really go and make it better than the usual there are really possibilities to do that and also if you have data import

you can really reduce on import time the transaction, we have the database where import 80,000 products with each several variants every week into the database so first time the import run the editors can't work while it import so now we are on a point where we

really reduce the import chain, the transaction time to adjust what we need to write the data and prepare the data before then start the transaction then the time was really small and so the editors now can work while the import is running

we mainly do internet actually big hospital systems so all our systems in Plone is about saving some data and then show it again and if we take the chain you describe

varnish and different things those are not so interesting because there are a lot of things we can't cache but if you take the internet only perspective do you have some extra tips if it's the internet only perspective the data basing comes very important if you have one user logged in

direct into the same instance all the time first time it loads the data and then it's cached there have lots of instances maybe depends on the system how many objects you want to have in memory cache use storage share the

unpackaged connection cache and I think then you have really to measure change parameters, measure again how it works and in internet scenarios it's much more difficult but the rule is more instances and stick really your users or your user groups depends on how your

internet is built up two different two clusters or one or a cluster of three instances or something like that we also tried something else I don't know if anybody else did it but trying rapid mq so we simply find the part of the calculations

that could take part after the after the save so we distribute that to another instance so when the user pushes saves then then of course is sent to some updated page but in the background another instance is calculating more outputs at the same time yeah it's a way of rapid mq is great for all the other stuff

it's also a thing we use for the import so you just fetched SQL data from an internal product database and made an external product view on it I think it's really fast and we use also a lot of Solr Solr helps a lot for this kind of scenarios

I think that's it, thanks very much Jens