I'm going to talk about debugging your Plone site. Though this will be more than just Plone, I'm going to cover the basic debugging process in general. Disclaimer, I do not plan on eating bugs like this cardinal in this picture. Don't be expecting that. You can find me online as CDW9 most places.

I am a Python developer at Six Feet Up and I'm also currently the Plone Foundation President. If you want to learn more about me, you can check out the Plone podcast that Six Feet Up puts out. I'm featured in the latest episode that was just released yesterday on a streaming service near you. Bugs happen. They can be discovered during

development through automated or manual testing and by your users in production. I'm going to cover basic debugging steps that can work on any website and then dive more specifically into debugging Plone. I've broken up the debugging process into

three main steps of reproduce the bug, find the cause, and then fix the bug. Finding the cause can sometimes be the most time-consuming part, especially if the issue is not immediately obvious. It sends you off looking for trying to figure out what piece of code is actually causing the problem.

The first step is to reproduce the bug. If the bug was found by someone else, the first step is to see it for yourself. This allows you to confirm the behavior and help you to make sure that the bug is actually fixed when you put in the code to fix it. Reproduce the bug. First of all, make sure it is reproducible.

It's possible it might only be a problem for the person that reported the bug, such as if it's a browser plugin. I've had it before where I was working on ads for a website, and then our QA engineer came back and said, well, the ads aren't showing, and it turned out it was a browser plugin that was blocking them. To go with that, it might depend on the browser,

if it's only happening in Safari. That goes with the next step of understand how it's reproducible. It might be something that is only happening in a specific browser. It might be something that is happening sporadically, like maybe it's due to slow page load. It could be the JavaScript ordering.

If you have some JavaScript that is asynchronous, that should be, you know, you need to make sure things are happening in a correct order. Could even be a specific instance. So like if your site is load balanced across multiple instances, maybe it's only one of those instances that's having a problem, in which case you would see it,

the problem on refresh sometimes, and then sometimes not. Also be careful because users or other developers, when they report the problem, they may make some assumptions. Don't mistake those for truth. Correlation does not imply causation. You know, they may come up with an idea of why something happened when that may not actually be the case.

If you can run the website locally, fix the bug this way. So let's say you see the problem on production and reproduce it on your local instance before fixing. Now, if you can't reproduce it on your local instance, well then, you know, check on production. There's a lot of going back and forth

with this whole process sometimes. But it is very important to reproduce it locally, because if you don't do that first, then you don't know if the code that you put in actually fixes it. You may start up your instance, write in code to fix it, and go looking, hey, the bug isn't happening. Well, it never actually was happening. So your code may not be correct.

And make sure that as you're testing locally, test against the copy of the data. If you are able to grab the data from production and use that locally, that's gonna be able to help you to debug the problem as well. You also don't want to go into bug fixing blind. Unfortunately, sometimes this is necessary where you can't reproduce a problem locally.

And so you write some code that hopefully will fix it, and you don't find out until after the release. So avoid that if you can, but it's not always easy to do. So once you have reproduced the bug, next step is to find the actual cause of the bug. Sometimes it's really easy to know what the problem is

as soon as you see it, and you don't have to do this step at all. But then sometimes you also need to go through a very thorough process of figuring out what is actually causing the bug. So the first place I would usually look would be the logs, or if your site is hooked up to Sentry, or look at the JavaScript console.

Keep in mind that whatever problem you're seeing on the website, it may not be an actual error being thrown on the website itself, but the logs may offer some help for this. Yeah, and not all issues throw an error in the logs. This is just the first place I usually go look

to get some information. Next would be figuring out if it's an issue with the data for the content. If you're using a content management system, is it a problem with something the user entered, or is it actually a problem with a code? So you can test the site with and without data and see if the bug still happens.

Knowing which one it is, whether it's data or if it's a code issue, is gonna help determine your path forward from this point. Next step would be to use the Python Debugger. Check out Philip Bowers' training on Monday, because he did a really in-depth training about using the Python Debugger,

and I learned some things from it. It was really awesome. I'll just note that if you're using Python 3.7 and up, you can put a breakpoint into your code called breakpoint. Otherwise, before Python 3.7, you have to do import pdb and pdb settrace.

Use the Web Inspector tools. You know, this goes back to checking the JavaScript console for errors. Even if you didn't do anything with the JavaScript, there might be something in the site that's causing problems, and it's preventing other JavaScript from loading or events from firing.

Think about, was this introduced by a recent code change? So if a release was done to the site in the last week, you know, that code should be suspicious. So review the recent commits that were included in that release. Also, clear the cache. Maybe caching is actually causing the problem. If the release updated dependencies,

check the issue trackers for those add-ons and also their changelogs. Also, don't rule out the possibility that the bug might be in the add-on itself or in core code even. So like if you are using Plone or Django,

you know, it may not be the custom code that you built on top of it. It could just be in the core code. So it's very likely that you will get stuck at some point while trying to find a bug. And when you feel that all is lost, here are some places that you can turn to.

The first option, rubber duck debugging. This is explaining the problem to a rubber duck. I know I've seen some pictures of some of you already that have a rubber duck sitting on your desk so that when you get stuck, you can explain the problem to the rubber duck. If you don't have a rubber duck, you can talk to a dog. If you have a dog, I don't recommend a cat

because they'll probably just walk away or some other random thing on your desk if you have something. But next would be to ask colleagues or to pair program. Don't spend hours banging your head against the wall. Do what you can to find clues and then turn to your peers for help.

And don't be afraid to ask for help. I hope you all work in an environment where everyone wants to help and isn't gonna look down on you for not knowing what the problem is here. I mean, we all miss things. Sometimes it's just a simple typo and having a second set of eyes is super helpful. And also maybe if you're completely stuck

that the other person you pair programming with may have had similar issues in the past and can provide guidance. If it's something that they've run into before, they may remember, oh yeah, all you have to do is this, or it might be looking up to see what they had done for that before. It's not always easy. And while you're pair programming, looking at the problem,

the other person may think of new things to try that you haven't tried out yet while trying to track down the bug. And then also you can send an email or a post on a community forum. You know, you don't get answers as quickly this way, but sometimes you'll find like you just start writing out the problem. And it's kind of the same thing

as explaining it to a reproductive, but this time it's just writing it instead of speaking it. You may start writing out the problem and figuring it out before you actually send the email or post on the forum. I have done this a few times myself. Lastly, what you can do is just walk away and come back to it later.

Do not underestimate this, especially when you get to the point that your brain is fried. You're so tired of looking at this, trying to figuring it out. Staring at the problem is not going to uncook your brain. So go take a walk, come back to it tomorrow, sleep on it. I mean, who knows? If it's something you're thinking about enough, you may wake up in the middle of the night

and think of something else to try. I have done that several times myself. So once you actually figure out what the cause of the bug is, the next step is to actually fix the bug or catch the bug as is the case with this fly catcher doing some fancy moves to catch a bug. Now, if you can see that in his beak, but there's a bug there.

So, you know, have an idea of how you're going to fix the bug, but it's not always easy to go through the process of doing a quick code fix that you can release right away. So think about, all right, is there a workaround that you can give the users for now? Is there something that you can provide

as an immediate help while you're going and doing other things of fixing? If the user did something wrong, as far as like the way they entered data or what they did in the site, is there something that you can put in to include in your fix so that it will expect that behavior or, you know, prevent the actual error from happening?

And it's best to do any fixes that you can in the code instead of expecting the users to do things the right way. Make sure that whatever fix you put in place disrupts the users as little as possible. You know, you don't want to expect the users to have to do something after the code fix is in place,

but do whatever you can in the code to be able to help them. Also, when you fix the bug, make sure you update the tests. This is going to help you to make sure that, you know, this bug doesn't happen as a regression in the future, and it's also going to make sure that you have complete code coverage.

And after you have fixed the bug, there's some follow-up that should go with that along with that as well. You know, do what you can to help your future self. So if there is an issue that is attached to this particular bug issue or a ticket or something, you know, make sure that's updated, especially with information about how you fixed it.

If you committed the code, make sure that there's a link to that commit in the issue. Also, update the forum post or your collaborators with your answer, especially on the forum post. You know, if you added a question, you may have gotten some responses that didn't help, or even if they did help,

make sure that you add a comment saying exactly what fixed it, because this is going to help lots of other people too that run into the same problem. If they come across that forum post, then they may find the answer. Also, if you did any pair programming with someone else, let that person know what the fix was, because, you know, as developers, we're very, very curious.

We want to fix the things. And so we also want to know what the fixes were if it was something that we helped you with on a problem. And lastly, you can submit a pull request. So if you did find that it was a problem in some core code, maybe you put in a workaround in your custom code, but also make sure to submit a pull request

to the core code as well. So now I'll talk more specifically about debugging in Plone. So this goes along with the steps that I've already outlined, but these are going to be some specific places that you can look inside of a Plone site that will help you figure out what is causing your bugs.

So first would be to check the error log. So you can see the errors that were logged in a site without logging into the server. This is in the ZMI and error under log. You can see there in the middle that certain exception types are ignored. So unauthorized, not found, redirect.

And in this case, you can see I commented out not found. So now when, if there are any 404s in the site, they get logged. You can see that along at the bottom there. So this can help you, you know, if you're trying to track something down. I have found that sometimes commenting out these ignored exception types can help, but not always.

Keep in mind here though, that the error log is only going to show the errors on the current instance. So if you have a prod site with multiple instances running that are load balanced, you know, you're only going to see the errors for the particular instance that you hit at that time. You know, if you're just going through the production URL.

And so as you hit refresh, you may see different errors. Or if you try to click on one, it's going to seem like it has disappeared once you click on it and get in there. So at this point, you can log directly into the server and actually look through all the logs of all of the instances. You can check the undo log.

This works best when it's a change that someone had made in the site recently. You know, if it was something that, you know, happened a week or two ago, don't even go to the undo log unless, you know, it's a site that no one touches very often. In older versions of SOAP, this is a tab that's at the top of the ZMI. And it changes with whatever, with the context.

So you can drill down further in the site and click undo and see the changes that happened at that level. But this is not in the case in SOAP 4. So you can see the path at the top of my screenshot here that if you go to control panel, which is, there's a link to this in the upper right corner of the ZMI now. Go to database and then main.

And there's the undo tab at the top to show you all the other things that have happened in the site. You can hit the server, a specific port directly, like a specific instance. And with our build outs, we have an instance debug instance available.

So this isn't running with a regular instances. It's just an extra one that's available that can help us to debug issues and in production if needed. Like I said earlier, you want to debug locally as much as possible, but sometimes you do have to do it on production. So here's an example where I have an instant debug script

that is running on port 8089. So I can SSH to that port directly, go to the site and then run instance debug in foreground mode, that is the FG. So that's gonna show everything output to the console of exactly what's going on.

And so you can even throw a PDB in some code as long as you don't restart the other instances at this time to then see what problems you're running into in the site. And you can view the site, like I have the link there. So in this case, it would be localhost colon 8089.

But even though it says localhost, don't be fooled. Remember that you are on production and don't go make changes, at least not ones that you don't intend to clean up later. You can export a section of the data as EXP. So especially if you're working

with a production site that is huge, like it has 50 gigabytes of data, you don't wanna pull that down to your local instance. But if you know that the bug is happening in one specific little part of Plone, you can export just that folder where you're having the problem and import it into your local instance. As long as the local instance, the add-ons, everything that's installed is the same

as the production site. So to do this from the ZMI, you can select the checkbox for the folder that you want to export. And at the bottom, there is an import export button, which it'll take you to this screen. So at the top, it has the export section. It's telling you, this is the folder you're going to export, and you can say, download to local machine.

Last I checked, this only worked with one folder at a time. So don't try to select multiple folders and do an export. I think you have to do one at a time. And then to import it into your local instance, you'll go to the build out inside of the var parts instance import folder.

Throw it in there. And then you'll go back into the ZMI on your local instance this time. Click that import export button again, and you'll have a dropdown that shows that ZXP that you put into that folder, and you can import it. And you don't have to import it into the same, it doesn't have to be at the same folder path as it was on production. You can throw it in anywhere.

Some add-ons that can help you with debugging. And once again, this is only for your local instance. Don't install these on production. There's Plonap debug toolbar that offers some context information through the web. Collective.recipe.omlet that can go inside

of your build out. And it's gonna provide inside of the parts folder an omelet that basically it just points to the eggs, all of the eggs that the site is using, the specific version that your site is using. So you can even open those up, look at the code, throw a PDB in there, it may even make some changes, but always remember to remove those changes

when you're done so that it doesn't affect the site later or other sites if you're using shared eggs. There is also a Plone.reload that allows you to reload the Python code through the web without having to restart the instance. And then products.pdb-debug-mode,

talk about this one more specifically. This offers a post-mortem debugger so that as you're running your Plone site locally, if there's an error with something, it's not just gonna throw it into the log, but if you're running the site in foreground mode, that's gonna put you right at a PDB prompt right where that error happened, which is super helpful for seeing everything in context

and figure out what the problem actually is. It also offers an add at PDB browser view. So if you're looking at a page and maybe there's not an error on the page, or if you just wanna see information about that particular object you're looking at, add add at PDB to the URL, and it's gonna put you in a break point in your console

so that at this point you can view that object so you'd be self.context and you can see everything that's attached to that object. And it's a full debugger too. So if you want to import the Plone API and do lots of other things just to test around, you can do all that right there.

In Plone 5, you can also add breakpoints to your templates. So that's this code that I have here. If you're using Python 3.7 and up, you can change that to calling breakpoint instead of importing PDB. But this works very similarly. Instead of self.context, it's gonna be that econtext variable that's there.

The econtext is a dictionary that's gonna have all of your templates variables. So if you have a bunch of variables defined within a template, put this breakpoint after that, and you'll be able to inspect what all of those variables are. If you are using Plone Classic,

this would be like if you have Barceloneta, a diazza theme installed. You can, I recommend disabling the theme. Maybe you're not sure where the bug is coming from. It could be something in your theme, you're not sure. So disable the theme. Go to the theming control panel found in site setup and switch the theme back to Barceloneta

by clicking the activate button and then see if you can still reproduce the problem. Also, in site setup is the resource registry control panel that you can turn on development mode. And this is going to turn off the minification and the combining of all of the JavaScript files.

And this can be helpful if you know there's a JavaScript error, but it's not clear what file it's in. This can help you figure that out. Also, if you can't figure out where a problem is coming from, start removing pieces. So like I said, on the last slide of removing the theme, next things that you can do,

test in the same site without data. Make sure you're testing against the same code base just without the production data in place. You could try to create a new Plone site at the same zoop root, especially if you don't have all of the add-ons installed. Maybe install add-ons one by one and see when the problem starts happening.

Can also test on demo.plone.org. I go here a lot. It's kept up to date with the latest version of Plone. So right now it's on 5.2. And soon there will be demos for Plone 6, both using the React front-end Volto and Plone Classic.

Also the portal catalog has helped me a lot. Not necessarily with finding buns, but like let's say I have a bug with a particular content type, and I don't know where all of those objects live in the site. I can go to the portal catalog, the indexes, and sort by portal type,

and then click the browse tab at the top like it's showing here. And so that's listing out where all of those objects are split up by content type. This has also been helpful for me like when I'm looking at upgrading a Plone site to Python 3, if the site uses Plone form gen forms.

I wanna know where all of those forms live in the sites and how many there are before I start doing a migration. And then also you can check online, like I mentioned earlier, posting in a forum for Plone that is community.plone.org. If you wanna look in the issue tracker, if you know exactly which add-on is causing problems,

you can go look at that specific issue tracker. Otherwise the general one for Plone is in products.cmf-plone. All right, so now I wanna give you some examples. So these are some actual issues that I have debugged recently, just to give you an idea of the process that I followed

to figure out what the causes were. So example number one, I don't have a whole lot of screenshots here, so hopefully you can picture my description. So we have a client that has an intranet that uses products.webserverauth to automatically log the users into Plone.

After upgrading from Plone 514 to 517, that automatic login stopped working and go to the Plone site and I would not have the edit bar, I just was not logged in. Whereas every other time I always was. One difficulty with this was that web server auth only works on the server, so I couldn't debug this one locally,

that made things a little bit more complicated. I didn't see anything in the web server auth repo reporting any issues, so I didn't know if that was the actual cause of the problem or what was going on. We spent some time figuring out which piece was no longer working and I didn't do all this by myself, but also my colleague Anthony

had done some investigating as well and he found that the remote under user header was missing. We thought maybe it was an issue with Apache that just wasn't bringing everything through and we don't have full control over the server that we are working on. So we thought, okay, maybe someone else made changes that is affecting this here.

Now, eventually we were able to confirm that the issue was related to the upgrade, it wasn't a problem with the server. Okay, next step is then determine if it's some custom code or if it was an add-on. We checked, like I said, we checked the add-on for issues, nothing had been reported, but it's also not used a whole lot anymore.

But I was convinced this was due to the upgrade. We didn't have any other custom changes as part of the release. So at this point it was like, okay, which version pin that changed is the one causing the problem? So I looked through all of the releases for all of the steps, Plone 515, 516, 517,

there was nothing in there that really jumped out at me, maybe I missed it, I'd have to go back and look now. So what it came down to was starting with the Plone 514 pins and gradually changing all of them to 517. And I did this, like just, I grabbed a chunk of pins at a time, move them over, check to see if the problem happened

until I finally narrowed it down to ZOPE. And once I knew that it was ZOPE, I went to the repository and I found this commit. Prevent header spoofing via underscore dash conflation. So there were security changes inside of ZOPE that are what actually broke the add-on.

So since web server auth depends on the remote under user header, like I said, it has an underscore in it, ZOPE was now ignoring those changes that web server auth was putting in in order to log people into the site. So my fix for this was to pin the version before this change. It's not always a good solution to use a version

that does not have security fixes, but we determined for this particular site, since it's only an internal internet that's only accessible by VPN, that it would be fine for this site. Example number two, snippets were not always loading in some cases. So we have a client that uses the add-on,

uwatch.snippets. This allows admins to add a reusable, what we call an HTML snippet, which is actually a custom content type on the site. It has some JavaScript code. So like admins can add this JavaScript code and then editors can insert the snippet using uwatch.snippets into a separate page.

So the editors aren't actually the ones putting in JavaScripts. They can't add anything malicious. And so the client reported, so here's an example on the department's outreach dual credit page. You can see under there at the bottom, there's some JavaScript code,

because this is what normally the snippets would turn this into the HTML, would render it, and a form should display here. But it wasn't in this case. And so what they reported to me was that the snippet was not loading on the default page of a, on the default view of a folder.

So that URL at the top, the dual dash credit, it was not loading there, but they did notice that if they went to the page directly, then it did load. So what they reported was that the snippets were not loading on the default views. I checked, I confirmed the behavior. I then tested locally.

I set up a folder that had a default view and the snippets were loading for me. I investigated a ton. I found, ruled out JavaScript issues. I was pretty sure this wasn't actually related to any JavaScript errors or anything. I did happen to notice though, that the snippets would load if I added a trailing slash to the end.

So, you know, with a couple of these, I thought, okay, maybe it has something to do with the way that the web server is set up. So I kicked the ticket over to one of our sysadmins to check things out. And he went in, he hit the port directly, and he saw the same behavior. Eventually though, I realized that I could reproduce the problem locally if I use the exact same ID for the folder,

which I hadn't done the first time. Because I was so focused on that being a problem with the default view of a folder, I didn't think it had anything to do with the ID. And that's where the trick was because the default views were not the problem in this case. So now at this point, I dug into the uwash.snippets code

to figure out how it does the transform. And I found this at the beginning of the transform function that renders the HTML. If self.requestPathInfo ends with edit, return result. And if you recall, the URL was dual dash credit. So in this case, that's true, it ends with edit.

And then it just returns the snippet without actually rendering it. So that was my actual problem. The fix then was to change this code so that it ends with slash edit because we don't want the snippets to render on the edit view, but on the actual view.

So that was the fix there. And example three, in this case, a form was not displaying on the navigation portlet. So in this case, the client reported the problem was that they had a folder, has a navigation portlet off to the right.

They had a couple of forms and one of the forms was displaying in the navigation portlet, but the other one was not for some reason. This one, I ended up being able to solve more easily just because of my experience and knowledge with Plone. But I found that this one was still very interesting to dig in and figure out.

So they had checked and saw, okay, the exclude from navigation option was not selected. So it should be displaying. I checked and looked, I saw that you have the same behavior with forms not showing up. I double-checked the exclude from nav was not selected. It's the middle checkbox there.

So at this point, I exported this section of the site as a zexp so I could work with that locally on my local instance. I thought maybe it was a problem with the navigation portlet itself. I was curious as to why it was thinking it shouldn't be displaying. So I put a PDB into the nav portlet code and dug around in there to figure out, okay, why is this not displaying?

And eventually found that it was getting a true value for exclude from navigation. And so now at this point, I looked at the object directly and saw, okay, this is the case, but it's not selected in the UI. And so this one took some historical knowledge of Plone to exactly understand what was happening.

And looking at the source of that form, you can kind of see that there. You can see the ID and name exclude from nav, but it's using camel case. At one point in Plone's history, they changed this property's value or ID

from the camel case to the snake case. And this form was still using the camel case. Also this, I didn't know to mention, this is a product that PloneFormGenForm that is still using archetypes, whereas everything else on the site was dexterity.

I did eventually realize that if you check the box for exclude from navigation on a PloneFormGenForm, it recognizes that and it will hide it. But when you uncheck it, it doesn't display it again. I haven't actually dug in to figure out exactly where that part isn't working, but I know that this is the problem.

And I was able to just manipulate the object from the command line to be able to change that value. So those are my examples. And that was my talk. So thank you all for joining. I hope you learned some good debugging tips.