Big Fat FastPlone - Scale up, speed up
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
Title |
| |
Title of Series | ||
Number of Parts | 45 | |
Author | ||
License | CC Attribution - NonCommercial 3.0 Unported: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
Identifiers | 10.5446/48047 (DOI) | |
Publisher | ||
Release Date | ||
Language | ||
Production Place | Bristol, UK |
Content Metadata
Subject Area | |
Genre |
Plone Conference 201440 / 45
1
2
10
11
17
19
20
27
31
39
00:00
Plane (geometry)Software developerSource codeAerodynamicsFluidInformationDefault (computer science)Scale (map)Service (economics)Set (mathematics)PurchasingFocus (optics)Virtual machineLibrary catalogPhysical systemInformationArtistic renderingDefault (computer science)DatabaseSystem administratorMultiplication signFunctional (mathematics)Software developerGoodness of fitCartesian coordinate systemCore dumpQuicksortOpen sourceTemplate (C++)Scaling (geometry)WebsiteSinc functionWeb pageImage registrationLecture/Conference
02:31
Computer hardwareCustomer relationship managementPlane (geometry)MechatronicsKnotHookingGoodness of fitHigh availabilityVirtual machineWebsiteData centerService (economics)Server (computing)Centralizer and normalizerBlock (periodic table)Computer hardwareCartesian coordinate systemDifferent (Kate Ryan album)PlanningLecture/ConferenceComputer animation
04:00
Execution unitPlane (geometry)DatabaseStructural loadVirtual machineInstance (computer science)Product (business)Digital photographyOpen setProxy serverCategory of beingCloningFlow separationTelecommunicationWeb 2.0LastprofilClassical physicsReverse engineeringLecture/Conference
04:58
Local GroupWeb portalProduct (business)Network-attached storageData storage deviceVirtual machineWebsiteSampling (statistics)Student's t-testCloningStudent's t-testUniverse (mathematics)Virtual machineProduct (business)LastprofilSoftwareRepresentation (politics)Design by contractFlow separationPlanningService (economics)WebsiteData storage deviceoutputMultiplication signWater vaporNetiquetteLine (geometry)Lecture/Conference
06:31
Cache (computing)Plane (geometry)DataflowProxy serverWeb browserDatabaseChainStructural loadData storage deviceDifferent (Kate Ryan album)Server (computing)Bargaining problemClient (computing)Service (economics)Installation artComputer hardwareSemiconductor memoryDatabaseProxy serverService (economics)Software testingCloningPhysical lawCartesian coordinate systemClient (computing)Process (computing)MiniDiscMetreData storage deviceConnected spaceAreaSoftwareMeasurementVirtual machineInstance (computer science)Insertion lossView (database)Different (Kate Ryan album)Natural languageSession Initiation ProtocolLastprofilBlock (periodic table)Multiplication signStaff (military)CASE <Informatik>Axiom of choiceServer (computing)Web browserCore dumpPoint cloudWeb 2.0Structural loadMereologyWritingThread (computing)Suite (music)BefehlsprozessorSocial classWeightLastteilungParameter (computer programming)BitGraph (mathematics)SAP Business ConnectorCache (computing)Computer hardwareChainLecture/Conference
11:06
Server (computing)Data storage deviceVirtual machineNetwork-attached storageLevel (video gaming)DatabaseFluidCodeEmulationClassical physicsHard disk driveBuffer solutionMiniDiscSlide ruleDefault (computer science)Cache (computing)Client (computing)Object (grammar)Instance (computer science)Office suiteData storage deviceVirtual machineDatabaseService (economics)Computer fileEmailWeb 2.0Multiplication signMechanism designServer (computing)MiniDiscFile systemAsynchronous Transfer ModeLastprofilReal number2 (number)Kernel (computing)Connected spaceReading (process)Medical imagingInstance (computer science)Hard disk driveSet (mathematics)SoftwareChainClassical physicsPhysical systemCuboidObject (grammar)Database transactionMathematicsAdaptive behaviorCore dumpCache (computing)Client (computing)Zoom lensBitInformation overloadFreewareBefehlsprozessorCloningProduct (business)Block (periodic table)QuicksortMenu (computing)Remote procedure callGoogolSpeech synthesisRight angleMonster groupRoutingUltraviolet photoelectron spectroscopyConnectivity (graph theory)Forcing (mathematics)SequelBit rateTime zoneLecture/Conference
17:06
Virtual machineServer (computing)Client (computing)MiniDiscCache (computing)Object (grammar)Instance (computer science)Data storage deviceThread (computing)WebsitePlane (geometry)Virtual realityDatabaseDifferent (Kate Ryan album)Instance (computer science)VolumenvisualisierungWebsiteInformation overloadData storage deviceObject (grammar)Mechanism designVirtual machineLimit (category theory)Semiconductor memoryConnected spaceCache (computing)Multiplication signThread (computing)Configuration spaceSynchronizationException handlingPhysical systemMiniDiscDatabaseNormal (geometry)BefehlsprozessorProduct (business)Core dump2 (number)NumberClient (computing)Point (geometry)MereologyLibrary catalogFreewareAsynchronous Transfer ModeRow (database)CloningBitComputer hardwareStructural loadDatabase transactionServer (computing)Arithmetic meanTask (computing)Latent heatDefault (computer science)Address spaceNetwork topologyObservational studyInverter (logic gate)Shared memoryDampingCartesian coordinate systemDirected graphSoftware testingAverageSequelOcean currentSource codeLecture/Conference
23:00
Cache (computing)Data storage deviceThread (computing)Object (grammar)Client (computing)Asynchronous Transfer ModeInformation securityInstance (computer science)Web browserConfiguration spaceInformationLibrary catalogCodeWeb pageDatabase transactionEmailQuery languageContent (media)WebsiteIntelIterationCartesian coordinate systemDatabaseLatent heatDependent and independent variablesInclusion mapFlow separationDatabase transactionRevision controlOpticsMultiplication signTunisRight angleLibrary catalogSemiconductor memoryOntologyRouting1 (number)Different (Kate Ryan album)Network topologyBit rateResultantOffice suiteWebsiteSign (mathematics)Object (grammar)Point (geometry)Machine visionPhysical systemSet (mathematics)Default (computer science)SpacetimeReal numberMusical ensembleComputer configurationArithmetic meanCondition numberOnline helpMedical imagingForcing (mathematics)Content (media)Information securityMetadataMultiplicationSheaf (mathematics)Configuration spaceType theorySynchronizationIterationInstance (computer science)WritingInterpreter (computing)Goodness of fitSubject indexingCache (computing)Mobile appTemplate (C++)Asynchronous Transfer ModeLevel (video gaming)Parameter (computer programming)Data storage deviceClassical physicsCodeRootComputer fileSAP Business ConnectorLecture/Conference
30:28
Library catalogObject (grammar)IterationFluidSystem programmingComputer networkCache (computing)MeasurementPhysical systemService (economics)CodeDatabaseParameter (computer programming)Product (business)WebsiteType theoryTemplate (C++)Configuration spaceNormal (geometry)Default (computer science)Task (computing)Content (media)Local GroupComplex (psychology)Proxy serverPoint (geometry)Instance (computer science)StatisticsIntranetExtranetVector spaceStructural loadDirected setMedical imagingServer (computing)WebsiteContent (media)Instance (computer science)Object (grammar)BitService (economics)Cache (computing)Web 2.0Attribute grammarVariable (mathematics)Computer fileLastprofilLastteilungThread (computing)Web pagePoint (geometry)Group actionStandard deviationLibrary catalogConfiguration spaceDifferent (Kate Ryan album)Real numberSlide ruleShared memoryMathematicsCartesian coordinate systemMereology2 (number)MetadataDirection (geometry)Server (computing)DatabaseCloningProjective planeResultantFormal languageType theoryWeightText editorCodeParameter (computer programming)Mobile appProxy serverLine (geometry)Classical physicsIterationGoodness of fitGraph (mathematics)Multiplication signBit rateSoftwareRule of inferenceSpacetimeMilitary baseComputer-assisted translationSocial classSet (mathematics)Storage area networkSource codeVideoconferencingArmINTEGRALUltraviolet photoelectron spectroscopyPhysical lawRight anglePlanningPlotterOntologySingle-precision floating-point formatFilm editingMedical imagingComputer configurationForcing (mathematics)RoboticsMeasurementLecture/Conference
37:31
Plane (geometry)Pointer (computer programming)Medical imagingComputer-assisted translationRight angleTunisTelecommunicationInformationGoodness of fitGroup actionPoint (geometry)Factory (trading post)MereologyNumbering schemeMultiplication signCloud computingDatabase transactionMathematicsAttribute grammarLogic gateNeuroinformatikOrbitProduct (business)DatabaseText editorChainInternetworkingDirection (geometry)Function (mathematics)WebsitePerspective (visual)Different (Kate Ryan album)Term (mathematics)Physical systemReal number1 (number)CloningPlotterArithmetic meanInstance (computer science)Connected spaceAuthorizationRoundness (object)Object (grammar)AverageWeb pageMeasurementSemiconductor memoryMessage passingSequelWindowCAN busEstimationMultilateration2 (number)AreaCalculationRule of inferenceOffice suiteSoftwareServer (computing)QuicksortContent (media)LastprofilClassical physicsView (database)MiniDiscBand matrixGene clusterData storage deviceShared memoryStructural loadCache (computing)Slide ruleType theoryForm (programming)Plug-in (computing)Process (computing)Lecture/Conference
Transcript: English(auto-generated)
00:00
Welcome to the second talk today, a technical topic I would say. It's about scaling up Plone or making Plone fast even in smaller setups. My name is Jens Klein, I'm located in Innsbruck, in Tyrol, in the Alps, Austria, developer
00:31
since Plone.1.0, also did core development this time and today.
00:40
Also I love open source software, so it's the reason maybe why Plone and the community behind is a really good example for open source. My company is Klein and partner KG and we are member and founder of the Blue Dynamics Alliance which does Plone also that long. I have here a little advertisement, in short we do a sprint in January in the Alps if
01:05
you like skiing and sprinting and good beer and a vibrant city in the middle of the Alps just join us, registration is open. But back to the problem, Plone is by default not that fast like any application
01:23
server, it's not different from Drupal or other system. Even maybe pyramid is faster but it has not this lot of functionality. So Plone scales great, you can just add machines, that's what system administrators
01:44
love but sometimes adding machines is not enough. And so we have other bottlenecks left. One bottleneck is the database, always, so loading stuff, data, pickles from the ZODB
02:05
needs to be optimised, then the rendering of the site itself is the bottleneck, page templates are usually slow, then searching the information you want to display is slow, a portal catalogue is relatively fast but you need to use it and so on.
02:28
And then you have maybe to talk to a third party service, that's a problem too. So you have to know your knobs and handles and that's what this talk is about, I want to give an overview of the knobs and handles, you can use and turn and press and where
02:48
you can hook in to make it even faster. It's in a submarine there, a Russian one, old one.
03:01
So I have picked out some customers with different set-ups to go through the talk. One customer, they are in central service for cultural institutions, in short NUCCU, Nieder Usterreich Kulturwirtschaft in German, so they have over 30 plan sites running mostly
03:26
for three or for two sites, they have a cultural, for cultural services, so the budget is not that high but enough to make it running good. They have own hardware and own data centre, so we are running it there with virtual machines
03:50
on this two bigger or three bigger servers and do high availability with pacemaker cluster resource manager, the block device application.
04:03
I apologise it's in German, but I have this picture not in English. The classical set-up is usually that you have, you can see it here, web server, reverse
04:20
proxy and caching proxy, a load balancer, a clone instance on several machines and So we are in this set-up, we did it a way that if one machine is for some reason gone, then the other machine can take over the whole load, the database, switches over
04:44
to the above machine or the other stuff switches down, but the clone instances then are only on one machine, one virtual machine, so you may not handle peak load with it but the usual load is fine. As a customer, using clone for the portals, product portals, B2B shops, externets,
05:10
customer communication, they have different set-ups, they have a hosting contracting partner doing the hosting management, but the services we put on it like clone, we
05:26
have to manage ourselves most of the time. Also, the storage is on a network attached storage, so we have the input I.O.
05:43
problems with this thing, because it's not that fast. Another one, smaller one, is a student representative body from the university of Graz, and they are having one site with several lineage subsides.
06:02
So at the beginning of the semester, the students are all going on the site and will have a peak load at the beginning and afterwards it's more flat. So it was really bad hardware, they have a plan for five years for the server, so it can't update it, and it's always virtual machines and completely overloaded
06:23
hosts, and we have to make it fast anyway on this stuff too. So, really different set-ups, and if you then look into clone and all the stuff behind, it looks like that. A lot of bricks and you can put them together in different ways, but how to
06:43
make it fast. A bit to the basics, because it's all the places you can hook in. It's a database itself that can be different databases, then you have a kind of client connection pool into the SOAP server.
07:03
Then you have the clone itself, so the application threads running, maybe, or you usually have more of them. Then you have the load balancer, to have the load on different machines. And finally, publishing front-end, like the Caching Proxy Varnish web server,
07:26
and finally the web browser, which is also part of this chain. So, usually we use a Zoo server, or write storage, and this write storage is on Postgres.
07:41
And always using block storage on NFS or NAS. So, the client connection is usually the different Zoo clients, or the storage adapters, which are in fact also Zoo clients, like the write storage adapter. The clones are the clones, as you know them,
08:01
and a load balancer we have in the past used Pound and in the last two years I think we switched to HAProxy. Caching Proxy Varnish, Squid is past, if you use Squid, better use Varnish, really, do yourself a favour.
08:24
Same with web server, I think Nginx is also the first class citizen and Apache is kind of past if it comes to the scenarios we have here. For web applications, maybe Apache has in other areas its field,
08:44
but I think Nginx is fast, better suited. And finally we have the web browsers, and you know, yeah, the browsers. And then all the clone instances are talking to third-party services like SAP Business Connector or whatever you have there in the world.
09:09
And this has to be fast too. And also we're using Memcached. Memcached is primary to cache stuff and is used by both the client connection pool and the clone.
09:25
And so you may need more than one Memcached instance to make it good. So you have to consider also how many CPU cores do I have, the ratio between used RAM and disk cache available, the IO on networks, the IO to the disk or to the NFS or whatever it is.
09:45
And so, yeah, a lot of stuff. And to get all this together, first thing you have to do is to monitor all you can do, all you can monitor on your servers. So Munin is our tool of choice there and all the others around.
10:05
With Munin you get nice graphs and you can see over time how the load, how the memory usage is, how the ratio between memory and disk caches, all the stuff, CPU load, peak loads, IO weight of the CPU. You get all the base parameters.
10:21
Then there's FIO, it's a little tiny Linux tool you can measure disk IO with and it's really good to get an idea how fast your disk IO is. Then you really want to know how Linux, we are using only Linux, maybe BSD is different,
10:40
I don't know, but Linux how it manages disk and RAM and it's all the caching stuff. And for sure you have to know on what hardware you are running and how you which your machines in the cloud or whatever these virtual machines are if you have some, but you have to know
11:03
what they are really doing, otherwise it can get really slow. So first we have now the database. At NURQ we are using a ZIO server on a replicated blob storage with DRBB
11:21
and also there are some other services replicated and the blob storage is also on the same machine and is also replicated. So if one machine fails, it fails over to the other and the whole switch takes under one second
11:41
so it's really fast. So if you unplug your machine it takes one second and everything is green again. ZOOM tool, we are using a storage in the beginning we had a really large database we did all the product imports there
12:00
so it was about 300 million objects in the database and the storage just managed fine and the blobs and all the data are on a NAS on a network attached storage over NFS. And in Graz we have only one machine
12:20
and we are using anyway real storage on there because it behaves a bit better than the ZO server for this low budget server for this overloaded CPU cores and so on.
12:41
And first things first maybe most of you know it but this is important never store blobs in the ZODB. So if you use Plone 4.3 out of the box and you use news items with images the images are in the ZODB change it, it's possible there are Google for it
13:01
there are menus how to do it use blobs storage and also for your add-ons check it, what they are doing with blobs what are the images really if you put blobs in the ZODB in any way even with real storage it's worse but don't do it.
13:23
So if you use a classical setup ZO server and blobs storage then you need fast IO to the hard disk and a ratio a good ratio to have your database cached in the disk cache
13:41
from the system the kernel cache. That's most important although there are not many settings on ZO server I think to really reduce some overhead it's really fast as it is and I think that's fine.
14:06
The blobs storage can be configured in different ways so it can go through the ZOOB connection or you can have shared blobs what we are doing is using NFS and shared blobs so that we mount the blobs storage on the clone nodes later on
14:20
and you can also mount it on the web server itself, maybe read only you don't want to write that and deliver the blobs directly their mechanisms with this collective accent file which sets the HTTP, Excel or the accent file header depending on the web server
14:41
and then the web server really delivers directly from the file system so blobs and that speeds it up a bit and reduces your network traffic inside because otherwise the blobs would be fetched over your network from the database through the clone to the other machine where the varnish is on
15:02
and so on, get all the chain down and if you can just tell Nginx get it directly from NFS you don't have it three times the traffic if you use the storage it's slightly different
15:21
it's the same but much more important is never stop blobs in the SQL DB maybe with Oracle I never tested it, it's possible I heard about it I don't know do not use MySQL if you can avoid it it works
15:41
read a bit faster and if you have concurrent writes you can have problems with it and if your database grows and you don't use the history free mode and you want to packet it the consumed size on the file stays the same
16:00
also ensure SANE and Disk and Storage IO check it use FIO this tool and I have Ram left, I talked about it so if you compare both Zio and the Rails storage adapters there is one big advantage on the
16:20
Rails storage side the Zio server has a connection to the Zio client and always tells the Zio client if some object is invalid or new transactions are available the Rails storage adapter polls it and the polling
16:42
times the delay between saying it's outdated can be configured so if you have high read size or instances configured for reading you can and you can live that delay between having the most recent data is 60 seconds
17:02
after pressing save you can have on peak load times where you have really lots of requests on your server you can have this delay available to not overload the database back end it helps a lot and on the Rails storage side
17:20
you have not this disk cache which in fact is also a memory cache because the system caches the disk part in memory and so we never using the disk cache and having only the memory cache by per thread and on Rails storage side you can have
17:41
it in memcached memcached server can be shared all the pickles loaded can be shared between all blown instances running on the same machine and so it's yeah it's a big advantage it reduces again load on the database
18:01
and the RAM cache of unpickled objects per connection the usual RAM cache where it says cache 30,000 objects can be then reduced so you can have more CO clients running doing parallel work on the same machine if you have enough CPU cores
18:21
and not memory is the limitation so short to the customers we have here 30 sides and it's a very classical configuration maybe most kind of default configuration
18:43
except that we have the DRDB DRBD and we have two threads usually sometimes four and 30 sides and 30 and 60 blown instances mirrored
19:00
on the different machines sometimes we have history free Rails storage history free means usually you have an undo in blown and the transactions are just appended in Rails storage or in SQL it is an insert but if you have search to history free modes
19:21
it's an update on the current row so the database grows not that much then but you have no undo in clone if you don't want the undo which does not work very well then it's okay
19:41
and it speeds up the things a bit again there are two virtual machines for the clones on each virtual machine 16 instances with two or four threads
20:02
and we have also worker instance asynchronous tasks like import database syncing with the product database very special customer specific stuff but we have also memcache installed
20:20
and share the connection cache and that speeds it up a lot without memcache it's way slower and if you have that many objects in a Rails storage I don't know what the limit is but the usually packing mechanism of Rails storage consumes lots of RAM
20:42
memory and then you have the problem you have a problem with packing because on your machine all memory is consumed and at some point it says I don't want anymore so if you really have this problem look at this Rails storage and let's go packaging on PyPy which solves this problem
21:02
also checks for 300 million objects normal packing mechanism estimated 300 days of packing that's a lot and now we are down to one day or eight hours or something like that I think it's a new one takes a while anyway but it's okay
21:21
it runs on the you can run it on any machine so it doesn't have to run on the same machine as the instances are running and our small setup which really has also high traffic on it and has to render lots of sites we have six instances
21:42
with one thread and a RAM cache of 30,000 objects and interesting the interesting thing is to reduce on this bad hardware it's better to reduce the number of instances and make them deliver fast
22:00
and increase the number of objects you have really in your memory cache unpickled between the six threads the unpickled parts in memcache again and you have really to try them how many objects do I have in database how many are loaded all the time
22:21
looks at the catalog all the catalog fits into this 30,000 objects and so on and so on and we then use it as a poll interval at 120 seconds 120 seconds means that if a request comes in on a machine then it takes up to
22:41
120 seconds it takes it from the memcache and if this time was passed then it asks the database is there something new that helps also a lot it's all about database and it's really on sync point where you can really speed up Plone on this side
23:01
yeah, but Plone this was all database it's all valid for any Zope it's also valid I think for pyramid applications running on geoDB I think substance D is one of them but it's all not Plone specific in what I know
23:21
but now what can we do about Plone itself and that's the problem we have a request, we have a response and they're not Ploned us so what is it classical configuration issues we can turn off debug mode
23:41
we have a security, logging critical deprecation warnings we have to configure on a default Plone 4.3, Plone app caching because it's not installed you have to install it and switch it on even if you don't use Varnish if you have a really small site it's a good idea because it speeds up syncs a lot
24:01
you can have then there's multiple Plone instances and you can configure even Plone to use memcache instead of ramcache and you may want to check JARN check interval which gives you a value you can you can
24:21
put into the configuration of ZOPE what's the check interval of the global interpreter lock is I think so this is all documented on Plone.org how to do it so it's nothing new just keep it in mind there's a good section on Plone.org
24:41
in the docs the default Plone is almost not a problem it just really works good also the caching parameters are working fine but if it comes to add-ons and I think all of you use add-ons in Plone you have to take care about them if you write custom code
25:02
noplone.memoize and use it never calculate or search things twice I often saw in code that you have a thing in a template like tell condition and ask a few dot results and then next is tell repeat few dot results
25:22
and so it fetches the results twice and if you can avoid this do a tell define before or just cache it with memoize if it's the results that you can use over several requests unmodified you can build up the same cache key
25:40
then use Plone.memoize to cache the stuff and don't calculate things twice also in Plone you can take care that not only results directly are cached but also the prepared data so you get something from the catalog and need to filter it again for some reason
26:00
or need to get some data from another place or you talk to a third party system like a SAP business connector or something strange you have no idea what it does this third party system but if you get data out of it, cache it just that makes things fast and those are the usual bottlenecks
26:21
if I go to third party add-ons to make them faster I find all this stuff in several places not cached yeah, I say again never store blobs in the UDB use the catalog and don't over use muta data so what I always see is that I put everything
26:41
to muta data and then if you search the catalog and wake up the muta data it's not much different than waking up the real object and because there's so much muta data in the catalog that it doesn't really make a difference it's crazy then a big problem in Plone
27:01
write conflicts and write conflicts are 90% of the time write conflict means that one transaction wants to write the same object, ZOID ZOID thing on the database level then another transaction wants to do so if you have two
27:21
writes and they don't collide, it doesn't matter but usually this happens in the catalog and the catalog consists out of B-trees B-trees are consisting out of this buckets where one bucket is one thing, one ZOID in the database if you have to think of portal type
27:41
and the portal type index and they have the same portal type saved twice and they want to write the same bucket it gets a wide conflict and write conflicts are really a big problem the first thing you can do about it is to reduce the time of the transaction if you can do it and you're fine with it and it's good for your setup
28:01
try to make it really fast so that the transaction is really fast but sometimes it doesn't help because you really need to prepare data and check things and so on and then it comes to the point where you say okay let's use something different than the catalog it's the only way, in my opinion
28:21
to get around it and we have with Solr very good experiences at this point and I can really recommend using Solr my big vision is that in Plone 6 we have no catalog anymore I'm just using Solr that would save us so many problems
28:45
yeah next problem you have lots of objects small Plone content it's easy to make it fast, but what about if you have hundreds of thousands of objects and I think
29:01
it's you can really store lots of objects it's no problem you can load them but again, if it comes to the catalog it's a bottleneck also, what can I do about it is collective Solr and we have the scenarios
29:20
and yeah if it's not that many objects you can put the catalog on an old mount point and give it a real large memory cache 200,000 objects while your usual other Plone, the root mount point has only 30,000 objects
29:42
and that would work wire, but at some point this doesn't work anymore and then you will have to use Solr now, in Plone 5 we have dexterity and now we're used to work with archetypes last years and
30:00
always there was a mantra saying don't wake up objects don't wake up objects wrong with dexterity? if you have a folder and iterate over the items and you have things like simple images of files in it
30:21
do it, iterate over it wake them up it's the same as waking up the metadata of the object because in dexterity the objects and the space used in the database is very small it works out fine, you can really wake them up it's an interesting thing, we really tried to make it with catalog or with direct
30:42
iteration and it works you can iterate over it and with dexterity if it's not something insane then it's really a small object you wake up and it's fine, you can wake it up but if you have archetype stuff really don't do it it just blows up takes lots of time
31:01
and if you have some custom dexterity type and it's all stored as attributes directly on the thing it may be also better to not wake it up, so it depends a bit on but if it's a classical standard clone up content types it's usually okay
31:25
yeah this adding of metadata to the catalog is really a problem I saw it and some projects we got from other companies and then I looked at everything it was put into metadata there was a bunch of metadata added
31:41
and there's also a thing where the catalog really blows up and gets big and also the loading of the metadata gets slow again and then if you can avoid adding metadata don't add it if you use archetypes you may want to do it and if your dexterity is possibly cheaper
32:01
don't wake up the objects and also the yeah, the third party services, the next one need to be cached, that's what I said again
32:21
and you need to also think about how failsafe are the services all the problems that you can really stop an instance or thread an instance, you block it because the service doesn't answer and it waits until I don't know, kind of 300 seconds it should wait too long and then the user tries again
32:41
and clicks again on it and then it blocks the next thread and at some point no thread is available and then it blows up you just have no threads left and your site says 500 service is available so if you have serious performance I would say
33:02
first thing you can do is measure so don't try to configure something around and get real first data out of it and then you have some tools I wrote I will put the slides on slide share or something like that and also the video, but
33:23
if you then change something, don't change more than one parameter I saw that people change four or five things and you have no idea what happens because change one thing, even experienced IT stuff I watched them really trying to change five things at once
33:42
and so I think I have to tell you to do it, not change one. Then next thing is on the front-end side after the application server, clone app caching I think there is a lot to say about clone app caching but if you have a plain clone false result
34:01
you just can use it, enable it and it works fine if you have custom add-ons you really need to want to configure it and it's documented well how to do it also on docs-blown-org and it's really the part you want to configure if you have own add-ons write your custom code
34:23
and with Varnish together with clone app caching are really our superheros here to make clone fast because it really makes it deliver the pages directly from Varnish and you can get really really high traffic on a clone site with it and have cache hit rates over 95
34:42
up to 98% if you do it right so if you monitor your Varnish with Munin you see the cache hit rate and on peak time it gets all faster so we have the problem on a low traffic time in the evening at 11pm for example
35:01
the site is slow and then at 11am everybody clicks on the site it's really fast so Varnish is great and one thing is this micro-caching what we did so we added caching rule that just caches stuff for one minute or for two minutes or five minutes depending on the site and this really
35:22
helps also to speed up the things on this high peak load to really have recent content maybe takes up to five minutes that it's on line but you have good caching experience and it really buffers the peak load we use
35:42
it on some turbulent site and it speeds up things you see it also on the Munin graph it's nice to watch how the peak goes up to 90% or something like that then we have load balancer
36:01
and what to say pound is old but you don't use it anymore on recent setups and JProxy is a new thing it's more complex to configure but you can do more things with it important point is session stickiness so you can use it to direct one user or one visitor or one language group to
36:21
a group of clone instances and that way this group of clone instances can better cache the data from the database because the cache really is filled on this part of the cluster with data used by this group of visitors so you can really cluster your groups
36:41
of visitors by language or by by customer type on one external net we have or by you can cluster by different editors or non-editors and so on can configure the instances different then we have the web server
37:02
it's just working good the only thing about the web server is that you set the proxy underscore stuff variables right this is what usually and all of this documentation same and you have the
37:22
possibility to use this collective extend file to directly deliver blobs and save some traffic on your network yeah that's oh yeah the cat image I think every talk needs a cat image, this is things called bobcat right looks in Germany
37:42
we have some, some kilometers south of us in the national park so I took that one yeah this is a chat on the blown documentation performance and tuning, I think it's very good there's lots of information and you can read further into that
38:00
and yeah, have you any questions? have we see a microphone or yeah, okay
38:22
probably a lot of us have begun deploying on systems that have SSDs for their drives in the past year or so and I mean when I've done that I've changed several things at once so I haven't really been able to figure out the differences and I wonder if you Jens or anybody else is developing a good
38:40
feel for the way you switching to an SSD changes the way you might balance your memory for efficiency SSD is the next thing that comes up but on our servers we don't have SSD we're almost on the network attached sort of stuff but I think with SSD it would need new measures
39:01
how it works this guy gets faster and things are different, right? I mean, I ended up running more instances with less cache than I ran before and it seemed right but since it was a new install and I changed several things at the same time I couldn't measure the differences
39:24
I was going to ask one thing we've noticed with dexterity is we're expecting it to be a lot faster than archetypes and in certain situations it can be about four times slower when you're trying to look up an attribute on dexterity because it has to work its way all the way through the schema
39:41
and compute where that attribute might be Do you know if there's any work on speeding that up at all? Yeah Okay Dexterity is a special topic maybe if anybody is interested in dexterity and what's wrong with dexterity I had some last two weeks
40:00
I really dived into it I also know because it's slow because if you have a factory and it has to look up the factory and all the stuff on a behavior then it really takes its time to get an attribute if you have direct attribute access if you have a main schema or no factory with the dexterity schema
40:21
you have direct access to the attribute and it's fast if you want to have it really fast you don't have to use this factory but it's also a disadvantage I think it's fast if you're looking for something that's in the schema but if the attribute is not in the schema like if it's a base attribute like some of the internal zope double under ones
40:41
then it can be really slow Yeah, that's a thing we need to solve but there are other things to solve in Plone.x too I was just wondering so you talked about how you measure the performance of your server you know you can use Munir to measure your disk IO
41:00
and all that sort of thing do you have a good way of measuring if you like the output of your site how many page hits they're serving or what sort of network bandwidth your visitors are getting as opposed to the output you're getting from your disks if you see what I mean the real sites we can solve
41:21
I didn't get the question right I think So it's easy to measure how much IO you're getting from your disks or whatever but in terms of how many pages per minute you're actually managing to serve or how much network bandwidth you're getting out to the users rather than from your NAS or whatever Yeah, it's difficult to measure we had a tool what was the name
41:41
it was a really crappy windows tool but it was good to do the job and I really need to look up the name and then there are several tools out where you can have this click passes through the site and also vary around and log in users and such things so if you really want to know
42:01
the behavior of your site on peak load you need to have this kind of measurement tools I think there are also cloud services around to measure this but you can't get this into Munin I think external measures and then you have to try how how does it work so
42:23
yeah, it's a different topic to get this data sorry? you can request per second and also Varnish tells you the Varnish Munin log tells you the request per second yes
42:42
device per second yes right all the Munin plugins are showing this information but if you really want to know how many requests in a certain scenario I can get out of my systems you need an external tool to penetrate it
43:02
yeah in some slides you told us about reducing the time of transaction my question is how to do that yeah it's primary to reduce the time on transaction on save of a classical
43:20
content type it's difficult but we have also scenarios where we have forms and then save the data ourselves and there you can really go and make it better than the usual there are really possibilities to do that and also if you have data import
43:40
you can really reduce on import time the transaction, we have the database where import 80,000 products with each several variants every week into the database so first time the import run the editors can't work while it import so now we are on a point where we
44:01
really reduce the import chain, the transaction time to adjust what we need to write the data and prepare the data before then start the transaction then the time was really small and so the editors now can work while the import is running
44:27
we mainly do internet actually big hospital systems so all our systems in Plone is about saving some data and then show it again and if we take the chain you describe
44:41
varnish and different things those are not so interesting because there are a lot of things we can't cache but if you take the internet only perspective do you have some extra tips if it's the internet only perspective the data basing comes very important if you have one user logged in
45:01
direct into the same instance all the time first time it loads the data and then it's cached there have lots of instances maybe depends on the system how many objects you want to have in memory cache use storage share the
45:21
unpackaged connection cache and I think then you have really to measure change parameters, measure again how it works and in internet scenarios it's much more difficult but the rule is more instances and stick really your users or your user groups depends on how your
45:42
internet is built up two different two clusters or one or a cluster of three instances or something like that we also tried something else I don't know if anybody else did it but trying rapid mq so we simply find the part of the calculations
46:00
that could take part after the after the save so we distribute that to another instance so when the user pushes saves then then of course is sent to some updated page but in the background another instance is calculating more outputs at the same time yeah it's a way of rapid mq is great for all the other stuff
46:22
it's also a thing we use for the import so you just fetched SQL data from an internal product database and made an external product view on it I think it's really fast and we use also a lot of Solr Solr helps a lot for this kind of scenarios
46:42
I think that's it, thanks very much Jens