The past, present, and future of how we test Plone
This is a modal window.
The media could not be loaded, either because the server or network failed or because the format is not supported.
Formal Metadata
| Title |
| |
| Title of Series | ||
| Number of Parts | 44 | |
| Author | ||
| License | CC Attribution 3.0 Germany: You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor. | |
| Identifiers | 10.5446/60225 (DOI) | |
| Publisher | ||
| Release Date | ||
| Language | ||
| Production Place | Namur, Belgium |
Content Metadata
| Subject Area | ||
| Genre | ||
| Abstract |
|
Plone Conference 202218 / 44
10
14
22
23
26
27
40
00:00
Statistical hypothesis testingStandard deviationStatistical hypothesis testingUnit testingPhysical systemStatistical hypothesis testingDebuggerRepresentational state transferCloningMathematicsSource codeOpen sourceSoftware developerContinuous integrationSoftwareDistribution (mathematics)Point (geometry)Order (biology)Group actionCodeLastteilungCode refactoringFront and back endsTerm (mathematics)Materialization (paranormal)Contrast (vision)Digital photographyComputer fontFlow separationProxy serverInformationMultiplication signMoment (mathematics)Web browserQuicksortControl flowAreaRoboticsView (database)Classical physicsDifferent (Kate Ryan album)BackupHeegaard splittingCASE <Informatik>Set theoryGoodness of fitSoftware bugBitTemplate (C++)Revision controlFreewareDot productSoftware repositoryCore dumpBasis <Mathematik>Directory serviceComputer architecturePrandtl numberPasswordEndliche ModelltheorieBuildingChainMobile appWrapper (data mining)Link (knot theory)Information technology consultingRight angleType theoryString (computer science)Social classVisualization (computer graphics)Web pageMedical imagingDifferenz <Mathematik>EstimatorHierarchy1 (number)Musical ensembleMixed realityGreatest elementSubsetInstance (computer science)Default (computer science)Error messageMereologyProduct (business)Text editorFluid staticsStatement (computer science)Computer fileFunction (mathematics)Repository (publishing)Integrated development environmentProcess (computing)Branch (computer science)ResultantComputer configurationParallel portSoftware frameworkComputer hardwareService (economics)Block (periodic table)File formatConnected spaceSuite (music)Message passingScripting languageMaxima and minimaFormal verificationLengthCartesian coordinate systemForm (programming)Decision theoryState of matterAtomic numberRootConfiguration spaceSchwerpunktsystemBeta functionLoop (music)Confidence intervalLimit (category theory)SynchronizationPolarization (waves)Linear regressionKey (cryptography)Functional (mathematics)Projective planeBookmark (World Wide Web)EmailElectronic mailing listMathematical analysisExterior algebraLibrary catalogExistenceTraffic reportingExecution unitLibrary (computing)Database transactionDatabaseWebsiteDependent and independent variablesCuboidNumerical analysisSlide ruleComputer animationLecture/Conference
00:42
Statistical hypothesis testingCodeLecture/Conference
01:11
CodeSoftwareStatistical hypothesis testingLinear regressionMultiplication signLecture/Conference
01:42
Physical systemControl flowStatistical hypothesis testingConfidence intervalPhysical systemComputer animationLecture/Conference
02:11
Statistical hypothesis testingState of matterRootStatistical hypothesis testingProper mapGoodness of fitMeeting/InterviewLecture/Conference
02:41
CloningChi-squared distributionStatistical hypothesis testingContinuous integrationPhysical systemStatistical hypothesis testingMultiplication signLecture/Conference
03:10
Statistical hypothesis testingEvent horizonStatistical hypothesis testingLecture/Conference
03:41
Statistical hypothesis testingPlane (geometry)Point (geometry)MathematicsLecture/ConferenceMeeting/Interview
04:13
Revision controlStatistical hypothesis testingDefault (computer science)Execution unitCodeLibrary (computing)Lecture/Conference
04:41
Statistical hypothesis testingPlane (geometry)Boom (sailing)Computer iconWeb pageDensity of statesTrigonometric functionsProduct (business)AreaStatistical hypothesis testingDatabaseFunctional (mathematics)Library catalogPoint (geometry)Computer animationLecture/Conference
05:15
Information securityCode refactoringBookmark (World Wide Web)Source codeCode refactoringLecture/ConferenceComputer animation
05:42
Statistical hypothesis testingPlane (geometry)Zoom lensDisintegrationStatistical hypothesis testingEmailElectronic mailing listMultiplication sign1 (number)Computer animationLecture/Conference
06:14
Statistical hypothesis testingDisintegrationTerm (mathematics)ExistenceDependent and independent variablesContinuous integrationExistenceComputer animationLecture/Conference
06:45
Statistical hypothesis testingSoftwareStatistical hypothesis testingCuboidStatistical hypothesis testingLecture/ConferenceComputer animation
07:20
Statistical hypothesis testingWebsiteLibrary (computing)Point (geometry)Mobile appLecture/ConferenceMeeting/Interview
07:59
Quicksort
08:27
Multiplication signCore dumpRevision controlGoodness of fitStatistical hypothesis testingComputer animationLecture/ConferenceMeeting/Interview
09:00
Statistical hypothesis testingMultiplication signContinuous integrationStatistical hypothesis testingConfiguration spaceComputer fileRepository (publishing)Lecture/Conference
09:30
Repository (publishing)QuicksortSource codeLecture/Conference
10:07
RobotGroup actionSoftware frameworkRoboticsStatistical hypothesis testingWeb browserQuicksortStatement (computer science)Physical systemFormal verificationLecture/ConferenceSource codeComputer animation
10:44
RobotSoftware developerCartesian coordinate systemScripting languageForm (programming)Web browserDifferent (Kate Ryan album)Set theoryStatistical hypothesis testingLecture/Conference
11:21
Connected spaceCore dumpSuite (music)QuicksortLecture/Conference
11:55
Key (cryptography)Branch (computer science)Revision controlStatistical hypothesis testingMeeting/Interview
12:29
QuicksortBitLecture/Conference
12:57
Group actionCapability Maturity Model IntegrationGame theoryOvalGamma functionGroup actionProjective planeLoop (music)Configuration spaceRepository (publishing)Statistical hypothesis testingTraffic reportingResultantSource codeMeeting/Interview
13:30
GEDCOMRoboticsExterior algebraSoftware frameworkTerm (mathematics)Statistical hypothesis testingWeb browserSource codeComputer animationLecture/Conference
14:03
CodeMathematical analysisOrder (biology)CodeRepository (publishing)File formatMathematical analysisSlide ruleQuicksortBitOcean currentProcess (computing)Lecture/Conference
14:32
MathematicsStatistical hypothesis testingNumerical analysisDifferent (Kate Ryan album)Right angleCodeMathematical analysisComputer animationDiagram
15:07
MathematicsPlane (geometry)Repository (publishing)MathematicsStatistical hypothesis testingLecture/ConferenceComputer animationDiagram
15:51
Branch (computer science)Right angleMeeting/Interview
16:18
Open setStatistical hypothesis testingUnit testingQuicksortCASE <Informatik>Lecture/ConferenceComputer animation
16:47
Reading (process)Open setCodeModule (mathematics)Statistical hypothesis testingRight angleSocial classStatement (computer science)Function (mathematics)BitStatistical hypothesis testingFile formatUnit testingIntegrated development environmentLecture/ConferenceComputer animation
17:33
Statistical hypothesis testingProjective planeQuicksortLecture/Conference
18:23
Group actionRoboticsMathematical analysisSoftware frameworkBlock (periodic table)Group actionRight angleComputer animationLecture/ConferenceMeeting/Interview
18:58
Software developerTerm (mathematics)Computer configurationCodeStatistical hypothesis testingLecture/Conference
19:33
Statistical hypothesis testingCAN busStatistical hypothesis testingSoftware developerParallel portRoboticsLecture/Conference
20:05
CAN busStatistical hypothesis testingComputer hardwareStatistical hypothesis testingCodeLecture/ConferenceMeeting/Interview
20:39
Decision theoryMessage passingLimit (category theory)Statistical hypothesis testingRight angleMultiplication signMaxima and minimaPasswordLecture/ConferenceMeeting/Interview
21:26
Statistical hypothesis testingRight angleDebuggerRoboticsClassical physicsGoodness of fitSet theoryLecture/Conference
21:56
Flow separationStatistical hypothesis testingRepresentational state transferClassical physicsFront and back endsRevision controlLengthLecture/Conference
22:34
Front and back endsCore dumpRevision controlControl flowLecture/ConferenceMeeting/Interview
23:05
Statistical hypothesis testingRobotStatistical hypothesis testingGreatest elementUnit testingStatistical hypothesis testingEstimatorHierarchyBit1 (number)Lecture/ConferenceMeeting/InterviewComputer animation
23:43
Statistical hypothesis testingStatistical hypothesis testingStatistical hypothesis testingAreaWeb pageContrast (vision)Lecture/Conference
24:13
Linear regressionStatistical hypothesis testingStatistical hypothesis testingVisualization (computer graphics)Linear regressionWeb pageMedical imagingDifferenz <Mathematik>Statistical hypothesis testingLecture/Conference
24:40
TypprüfungMathematical analysisAsynchronous Transfer ModeCloningCodeMathematical analysisMultiplication signComputer clusterType theoryText editorLecture/Conference
25:06
CodeType theoryMultiplication signPoint (geometry)Statistical hypothesis testingLecture/ConferenceMeeting/Interview
25:36
QuicksortProduct (business)CodePhysical systemError messageLecture/Conference
26:07
Correlation and dependenceDefault (computer science)Tracing (software)Medical imagingError messageLecture/ConferenceMeeting/Interview
26:32
Computer networkMultiplication signPhysical systemView (database)Web browserQuicksortFront and back endsComputer animation
26:58
LastteilungControl flowProxy serverFront and back endsDifferent (Kate Ryan album)InformationLecture/Conference
27:32
Moment (mathematics)Front and back endsFlow separationSoftware developerBuildingClassical physicsRepresentational state transferHeegaard splittingSoftware bugControl flowStatistical hypothesis testingLecture/ConferenceMeeting/Interview
28:32
Classical physicsStatistical hypothesis testingError messageMeeting/Interview
29:05
Point (geometry)Statistical hypothesis testingDifferent (Kate Ryan album)Source codeDirectory serviceLecture/ConferenceMeeting/Interview
29:45
Ring (mathematics)Plane (geometry)Statistical hypothesis testingFlow separationCore dumpMultiplication signLecture/Conference
30:11
Multiplication signCore dumpBasis <Mathematik>FreewareOrder (biology)Group actionLecture/Conference
30:53
Goodness of fitGroup actionPoint (geometry)Dot productLecture/ConferenceMeeting/Interview
31:22
Commodore VIC-20Software repositoryBitInformation technology consultingSoftware bugLecture/ConferenceMeeting/Interview
32:05
Computer architectureAreaEndliche ModelltheorieChainMobile appBitLecture/ConferenceMeeting/Interview
33:08
Plane (geometry)Heat transferComputer architectureMoment (mathematics)Game controllerInstance (computer science)Lecture/ConferenceMeeting/Interview
33:35
Maxima and minimaTurtle graphicsMereologyLecture/ConferenceComputer animation
Transcript: English(auto-generated)
00:00
where there's not one right way to do testing. But let's talk a little bit about why you might wanna do some kind of testing. Here's a test. We've got function, return something. We've got the test function that runs that and then makes an assertion to see if it's what we expected.
00:22
And then you typically have a test runner, in this case I'm running pytest on the file and it finds all the tests, runs all of them, makes sure that everything's good. There's a saying you may hear in the polling community, untested code is broken code. And that comes, I think, from a talk that Martin Aspelli
00:43
and Philipp von Weidershausen gave in 2007 in Naples. And they were talking about how you can use tests to just help avoid making silly mistakes, but also to help you find design problems. You write the test, you realize oh this code
01:01
doesn't maybe work the way that's optimal so then I have to rethink things. Unfortunately, untested code is broken code. Tested code is broken code too. You're never going to anticipate all the different ways that your software could be used. So you can write the tests that you write,
01:21
but even if they all pass, maybe something else is suboptimal. So you have the sync, it's passing all the functional tests, but clearly something is not quite right. So if that's the case, why do we still spend time writing tests? We do it to avoid regressions.
01:42
If you find a bug, you write a test for it, then you know as long as the tests are being run all the time, you can't break that thing again without realizing it. We write tests to improve confidence when you're changing the system. So you need to do an upgrade
02:01
from Python two to Python three. It's a lot easier to know you got it right if you have some tests, so you can just run them before the system's working, you run them after some things break, you fix those things, and then you know in the end state that everything that was working before is still at least the things
02:20
that you're testing are the same. But really at the root of it, we write tests to minimize the chance of being ashamed. We have pride in the thing that we built, and we don't like it when it doesn't work properly. So tests help us make sure that it's good.
02:42
And also, to minimize the chance of being woken up in the middle of the night, I don't like that. But there's this other thing, you can't just write the tests, you have to run them also. So usually you can do that manually, that's good,
03:01
but usually we wanna have automated tests, continuous integration system every time that you're checking in something to GitHub, making a change, the tests automatically run and tell you so you don't have to remember to do that. All right, so let's look back.
03:22
Some of this history is from before I started with Plone, so I was kinda curious to see when did Plone get its first tests. So I went to look, and of course you pull up Git, you look at Git blame, Git history, and that only gets you so far because you run into this event horizon
03:43
where CMF Plone got moved from Plone or vice versa. So you can't follow that back. But I went back to SourceForge, and I went back and found the earliest tarball of Plone. It was CMF Plone 1.0 beta two dot tar dot gz, and I downloaded it, took a look,
04:02
and found this in a change log. 2002, October, Dreamcatcher's Sydney De Silva. I guess we were using CVS at that point for version control, and he was copying over some tests from CMF default.
04:21
So Zope Community was doing tests before Plone, but pretty early on, this is like the first year or so of Plone's existence, it had tests. And they looked like this. Well, we weren't using VS code then, but we have, it's based on the Python unit tests library.
04:43
It's using this transactional tests thing, so I guess each test is going to roll back the database before it runs. It's setting up a Plone site, and then testing creation of a member. So we call the add member function and then make some assertions about it.
05:00
So very first Plone test was this one, and then some tests related to the catalog and making sure that when items got indexed, you could search for them. Also, notice the left panel, that was all the packages that were in Plone at that point. I mean, of course, there was Zope and CMF as well,
05:21
but DC Workflow, CMF Plone, I don't know what active pack was or formulator, but pretty small. Also, my favorite source code comment in all of Plone was already there. It says, this should not make it into 1.0. Refactor me, my maker was tired.
05:41
It's still there. So at some point, we started running the tests automatically, and I found some evidence of this starting in 2004. There was a mailing list called Plone tests, and every time the tests ran, it would just sort of email the log there.
06:02
So this was one of the first ones in like January 2004. But also, we might have had CI earlier, I'm not really sure. I found this discussion from 2010. Balazs was saying, just to praise the Plone people,
06:21
continuous integration becomes popular. I think we were doing it long before the term existed. As a matter of fact, we should put in marketing materials, Plone applies CI since his birth, since it seems more meaningful today. And of course, my snarky response to that was, yeah, we've had CI for years, and this year, we actually got the builds passing, which I think they were broken for quite a while.
06:46
I started with Plone in 2007. At that point, we were using a thing called Buildbot. It's a CI software written in Python. It still exists today, actually. As you can see, it was very fancy at that point, the way it looked. I think somebody at Yarn, which was a Plone company,
07:02
had just a box sitting on their desk, and it was running the tests. But really, I think the golden age of Plone testing happened in the 2009 to 2013 area. There was just a lot of activity. There was a testing and CI team
07:22
that was active at that time. One of the things that happened was, before this, we had these base layers for writing tests. So if I want to write a test against Zope, I use the Zope layer. If I want to write a test against Plone, there was this thing called PloneTestCase that would take care of setting up the site
07:40
and installing things, so you had a stable starting point for your test. Martin Aspelli said, this isn't good enough. I'm gonna create this whole new library called Plone App Testing. That's what we use today, and it gave a lot more flexibility. Now, instead of just one Plone layer, you get what you get.
08:01
Hopefully, it's what you need. Now, each package can define its own layer that has the base Plone, but also itself, that package installed, and maybe some other things. Martin's work was based on a thing that Ross Patterson had created called Collective.TestCaseLayer that sort of paved the way to doing that.
08:24
In about 2010, we got Jenkins. Actually, it was called Hudson then, and then Oracle bought it, and there was some licensing stuff, and the community forked it as Jenkins. So this is still where we run a lot of our continuous integration today.
08:41
We've got builds in each version of Python that run every time there's a commit to one of the core Plone packages. We also have builds. If somebody's making a pull request, we can build that pull request, make sure it's good before it gets merged. I mentioned there's a testing and CI team
09:01
that got started around this time. I think that was team was doing, but there were lots of other people talking about testing too. In 2012, Travis CI came along, and this was really the first I had seen of this idea of continuous integration that's driven by a configuration file
09:22
in your repository. So you create a .travisci.yaml file that says run these things when there's a commit, and then you would tell Travis, hey, turn it on for this repository. It would start doing its thing based on that file.
09:41
That's really common now. We have GitHub actions, GitLab CI, things that are source code driven like that, but this was sort of the first I had seen that. Doman Kojar brought that up with the community, and as a result, a lot of add-ons started using Travis.
10:02
At first, they were very friendly to open source, gave lots of free builds and stuff. Eventually, they became less friendly, so mostly people have switched over to GitHub actions these days or other things. We also got Robot Framework as our first end-to-end browser testing solution.
10:21
So you could write sort of a spec for the end user, what behavior they're supposed to see, and then each of these statements got translated into actual commands that would run in a real browser using Selenium. So now we have full verification that the system is actually doing what it's supposed to.
10:43
There's actually quite a history of this sort of testing that's connected to Plone. I don't know if you know, Jason Huggins was a software developer. He was working in the early 2000s at a startup in Chicago, I think called ThoughtWorks,
11:00
and they had some needs where they had this application that was in Zope or Plone, I'm not sure which, and they needed to script it to fill out forms on the page or whatever. So that led to Jason creating the scripts that became Selenium, which has then become really the foundation for a lot of browser testing
11:21
in a lot of different settings. So there's a Plone connection there. In 2013, there was a sprint that Eric Steele and Timo and Ramon, I think, maybe some other people, I don't know, created Mr. Roboto. We had this problem where the core of Plone
11:42
is made up of like 100 different packages, and so whenever something is changed in one package, we actually wanna run the full suite of tests, which is located, well, we have this repository called buildout.coredev, and that's sort of the central thing that brings all the packages together and runs the tests.
12:02
So Mr. Roboto takes care of a couple pieces of key automation. One of them is if I make a commit to, say, Plone.api, and it's the main branch of Plone.api, Mr. Roboto gets notified of that, it checks which versions of Plone is this branch used in,
12:22
what versions of Python do I need to test on, and then triggers the builds in Jenkins. Mr. Roboto also made it possible for us to have this, some of the things you may have seen if you've been working on pull requests where you open the pull request and then it comments
12:40
and says, hey, thanks for contributing. I'm gonna check the contributor agreement. I'm going to tell you you can run this command to start the test, that sort of thing. More recently, I think there's been a little bit less innovation. Maybe I've just been out of the loop a little bit.
13:02
2018, GitHub started their GitHub action service, and so this is a really useful thing that a lot of Add-ons and projects are using now. Same idea as Travis CI. You have configuration in the repository that says when there's a commit, run this thing,
13:20
report the result. Also, recently we have Volto, and we have a JavaScript front end, so we're testing that also. We have Cypress, which is sort of like an alternative to robot framework in terms of how we're using it. You write, again, a spec for what the behavior
13:44
is supposed to be, your acceptance tests, and then it actually runs in the browser. Yeah, yeah, so we have tests for Volto as well.
14:00
And then maybe one of the latest things I've been seeing happening is starting to add these code analysis tools to our repository so that we're checking that the formatting is consistent, that you aren't making just really silly mistakes like using a variable that wasn't defined, that kind of thing, and we have some tools now where it's pretty easy to add those checks
14:20
to a new repository. So I've just got one slide here to sort of recap what our current process is, and then I'll talk a little bit about where we could go from here. If you're contributing to Plone, you go to GitHub,
14:40
you make your changes, you make a pull request, that triggers a number of different checks. We make sure that you have signed a contributor agreement, you're only allowed to contribute to Plone if you've signed over your rights to the code to the foundation. We do these linting and code analysis checks. We check to make sure that the PR has a changelog
15:01
so that we can easily look back and see what has happened. I missed a really important one. Anybody? Steve? If it's a repository that has docs in it, we build the docs too.
15:22
And then we've got the tests. The tests are not triggered automatically right when the PR opens, because maybe you're working on a change that needs to touch 10 different repositories, so you need to go create 10 different PRs, and then you can go to Jenkins and run them all together. But if you're just got a standalone change in one PR,
15:43
then you can just say that in a comment, Jenkins Plone org, please run jobs, and then it'll go ahead and trigger the builds in Jenkins. In the end, everything's green, then the PR can be merged. And once it's merged, that then triggers some other builds, because we want to make sure
16:01
on the main branch that everything is still good. Okay, so, okay. As you can see, the community has quite a history of innovating, building the right tools, so let's imagine where we could go from here. I don't necessarily have the right answers.
16:21
Mostly I'm going to be asking a lot of questions, things to think about or work on, so if you see something that strikes your fancy and that you want to pick up at the sprint, I'd be happy to talk about it further. Are we using the right tools? Most of the Python community uses Pytest.
16:40
We use the Zope test runner. This is sort of an example showing on the left what a Python unit test case looks like, and then on the right with Pytest, you can see it can be simpler. You don't necessarily need to write a test class. You can just use the cert statements and you'll still get useful output in the end.
17:02
It has this feature called fixtures. It's a little bit like Zope layers to provide the environment in which your test runs, but it's more flexible. You can mix and match different fixtures. Might be interesting to explore using Pytest with Plone. And in fact, it's pretty easy to do that.
17:22
I thought for a long time, oh, if I want to use Pytest, I need to rewrite all my tests first. That sounds like a ton of work. No, Pytest will discover your tests in unit test format and run them as they are. Just works. There's one thing, if you are using Zope layers, like we probably are,
17:41
you just need to install the goset.pytestlayer package, and then it turns the Zope layers into Pytest fixtures and the test will just run. So I've been trying this out on a couple add-ons and projects and so far, mostly is working pretty well. So I'm sort of excited to start writing tests
18:02
in the Pytest style when I can. But it might be, we need to figure out then what that looks like for Plone Core because we would need to decide, yes, we can switch over to run with Pytest instead of the Zope test runner and still have all the same tests and everything.
18:20
We can take a look at what the best tool is for doing browser tests. I mentioned Selenium robot framework and Cypress before. There's also a new kid on the block Playwright that I've been hearing good things about. I haven't done the full analysis of when it makes sense to use one or the other, so somebody can take a look at that.
18:41
I think we should look at moving our automation from Jenkins to GitHub Actions. It's just going to be easier for the community to maintain over time. Right now, there's like one or two people that can fix issues that come up on Jenkins and they do come up. Thank you, Jill Farkata, for keeping it going for us.
19:03
But if Microsoft is offering this service, maybe we should take advantage of it. Also being mindful that Microsoft has our repository, the S code. We're handing over a lot of our development tooling to Microsoft and want to make sure
19:21
that we have the road off if we need it, if they change the licensing terms or something. But for now, it might be a good option. Can we make the tests go faster? Right now, it takes like 30 to 40 minutes to run the tests on Jenkins. If we make the tests go faster,
19:41
we can make our developers go faster. We should take a look at whether there's any easy wins. I tried, I got a M1 MacBook Pro. I ran the test here, turned on parallelization to use all the cores. This wasn't with the robot tests, but it ran all the tests in about four minutes.
20:02
Do we just need to get some Mac Minis and have them running the test somewhere? Maybe just throwing hardware at it is a way to make some improvements. Also, go see, Nates has a talk tomorrow about a tool he's working on to help measure the performance of tests that I think looks really interesting.
20:25
Maybe we should put more of our repositories together. This is something we've explored in the past. There's some things that doesn't make sense for, things that are also used outside of Plone, but there's not a whole lot of reason that we need to have all of our code
20:42
in separate repositories, even if we're releasing it as separate packages. We can make that decision separately, and it might make things a lot easier if, for example, a week or two ago, I was changing the minimum password length. So I found Plone Pass where the limit was.
21:02
I changed it. Tests broke across 15 different packages because there were all these tests that had hard-coded the password, and so I had to go through and open a bunch of PRs. If we had a monorepo, I could have made one PR. We wouldn't need all of the tooling with Mr. Roboto to make sure that the right builds
21:20
are getting triggered at the right time. I think it's a good time to look at this again. Are we testing the right Plone? We've got multiple front ends. Which one are the tests running against? We've got the robot tests against Classic. We've got Cypress tests against Volto. Do they cover the same features? I don't know.
21:40
Does anybody know? We've got multiple back ends, REST API, Giussina. Rob's gonna be talking about NIC. Right now we have a pretty good set of tests in Plone.restapi for the REST API, but you can only run it if you've installed
22:01
the Classic Plone back end with the REST API. Wouldn't it be cool if we had a spec that you could run just over HTTP, separate process, and you could test NIC, you could test Giussina, and you could test REST API and make sure that it's behaving the way that Plone back ends are supposed to behave.
22:22
Which version are we testing? There's this issue that happened recently, actually with the password length. I went and made all those PRs. We made a new release of the back end. And then we tried to run the Volto test against it, and they broke because the Volto also had some hard-coded copies of the old password in it.
22:43
And what it made me realize is that we aren't testing the back end of Plone against Volto until it's too late, right? So we need some sort of build whenever a commit is made to core that runs Volto against the unreleased version
23:01
of the back end and make sure that we find out early if we're breaking something. And we could look at the tests themselves. Every talk on testing has the testing pyramid. You've got lots of unit tests at the bottom. They run fast. You've got acceptance tests, but they tend to be slower,
23:20
so you want fewer of them. And I did some really rough estimates of what we've got in Plone right now, and it roughly follows that hierarchy. I wanted to do more analysis, and I actually found it a little bit difficult to know how to look in more depth at what our tests are. Which ones are worth running? Are there tests that just are pointless
23:42
because they're either never failing or because they're only testing something that's, they're testing something that's only used by the test now. Finally, I think there's some new areas
24:01
we could look into. We could do more accessibility testing. We have one accessibility test in Volto that uses the Axe tool, checks things like color contrast and so forth. But we could run that against more pages. We could test the classic UI as well. Visual regression testing.
24:20
This is where you have a snapshot of what the pages look like, and then you can show a visual, an image that has a diff. Not necessarily something to break the build, but just to be aware if you're accidentally changing some CSS here that also affects the CSS somewhere else, you wanna find out about that. Back in the day, we had these performance tests
24:42
that would tell us if we were making a clone slower, and it's been a long time since we had something like that. I think there's probably better tooling available for that right now. I won't say too much about type checking. It's a way that you can annotate your code with hints about this is a string,
25:02
this is this class, et cetera, and then your editor can do static analysis and point out mistakes before you're running the code. If you're using VS Code, go turn this setting on. Well, maybe. You'll turn it on, and then you'll start to see red squigglies everywhere. And maybe you like that and maybe you don't.
25:20
Some of them are false positives. But the point is, VS Code has Microsoft's proprietary type checker called Pylance, and it can do a lot of inferring what the types of things are, even if you haven't taken the time to write any of the hints. So it helps me catch things like, oh, I'm using this variable,
25:40
assuming it can never be none, but actually it can, that sort of thing. I don't always test my code, but when I do, I do it in production. We can work on our tooling. You're never gonna catch everything, right? So we can work on our tooling for once code is in production, errors are gonna happen.
26:01
How can we make sure that we're finding out about those, the sort of thing that Sentry does, where errors get sent to a central system and you can review them? But maybe we need something that is not Sentry so that it's easier to host or not deal with GDPR complications.
26:20
Wouldn't it be cool if that was something that came by default when you deployed Plone with the official Docker images and you just had a place you could go to see the errors that were happening? And also distributed tracing. This is when you've got a system that's made up of many different parts, having tooling to help you see what's happening,
26:40
where, at what time across all the different systems. This is not that. This is the lighthouse view in my browser showing all the different requests that are happening. But I put it here because it's sort of like what I would like to have for the backend also. So the big blue bar up there, that's what happened on the backend. And right now we can't break that down at all
27:01
to see like this went through the load balancer and Volto proxy and the backend and then trigger these different sub requests. It'd be cool if we had tooling that was recording that information and making it available to us. So I think I'm about out of time.
27:21
Thank you for being here and please feel free to ask questions. Other questions? Yes.
27:41
So one of the subjects I think you touched but another issue we have at the moment is that we have the Volto frontend separate but the backend still also has all the classical UI stuff that that means that sometimes a classical UI bug will break the backend and then will not work anymore with Volto
28:00
when it actually has nothing to do with Volto. And when you're specializing in either classical UI or Volto, and in this case, a Volto frontend developer has to cease the build breaking and doesn't know that it doesn't know his way or her way around in classic UI and then has to ask other people first, like can you check if this is actually a backend backend or a classic backend backend?
28:21
So another thing there which you already touched on is like splitting the classic UI tests from the backend backend blown REST API checks. Yeah, I'm trying to think because I mean, depends on where the error is, right? If it's in the classic UI,
28:42
then it's probably not affecting Volto but if it's in the REST API, then it probably is. Maybe I didn't understand the scenario. I think Fred is hinting that the classic UI
29:01
should be split into its own thing, let's say. If we had the tests for the classic UI separate, it'd be easier to tell whether we broke that or all of them. Yeah, okay, fair point. You haven't talked about local testing, so is there something like talks or something
29:22
that you would recommend to others? I've used different things. I guess the important thing is to be running the tests the same way locally that you are in CI just to minimize differences there but I don't have a specific recommendation. I have a second question.
29:41
You had the point with pytest moving all the source code into the source directory. What's about moving the tests into a separate test directory? You can do that also. Yeah, actually that goes along with that change, yeah.
30:05
Do you think we can use pytest as the fall for new add-ons even without doing that in the core because every time we mention pytest in the core, actually every time we mention pytest, people that need to work on the core
30:20
on a daily basis they jump like 10 meters high and start yelling no, no, no. I mean, yeah, there would be some things that we would need to figure out in order to make it work well with core. I think add-ons are the place to start and I think let's do it for several specific add-ons before we make it the template but yeah, it might work.
30:47
Just want to mention that GitHub actually allows you to have your own runners for GitHub actions and they're free and the idea of having a couple of Mac Minis set up as GitHub action runners is a very good idea.
31:01
Yeah, good point. I knew about that feature but I had not connected the dots there.
31:22
Yeah, but we have the same problem with Jenkins. In the end, we need to trust the people that we bring into the community. And when you're talking about monorepo, what's your opinion about merging all the blown dot packages in one package? All right, merge, go further than just merging the repo
31:43
but also release them all together. I think that's a little bit more controversial because there's a lot of consultancies that it's nice to be able to fix this one small bug in this small package and then make a new release and just override that package. We wouldn't be able to do that anymore.
32:01
Maybe it still makes sense, so we'll discuss this. This is about monorepo. You have to think about that we give up something that we tried now to clean up is that the dependency chain inside this blown app packages is kind of broken and we did some effort to clean it up.
32:20
It's not finished. But if you have all in the monorepos and it happens that everything depends on everything and even this dependency chains break more. And so I'm a bit against having a, I like the idea that we consolidate packages and do not have every single package with every micro features in our package.
32:41
But overall, I would really like to think about architecture and not about just monorepo as a thing that solves it all, that's wrong. Because it's more about really getting into it, solving it in a way that you define an architecture, say what are our main areas we work on? What is our mental model of our architecture?
33:01
And then inside this mental model of architecture, we can think about merging stuff together. Yeah, I mean, take the suggestion of a monorepo as let's have fewer repos, not necessarily just one. I think thinking carefully about the layers and the architecture makes it. And at the moment we have this, like we have everything in control panels,
33:21
it's control panel instead of having the control panels where they belong to and that makes it different to work on this, for instance. We can't get into the details now of where you have to finish up. That's what we need to discuss maybe somewhere as a strategic thing we need to discuss. Let's continue that discussion later, yeah. Okay, thank you. So, for the questions, this is the end.
33:43
Thank you very much. Can I quickly help you here and put this on there? Take care of this part.