Hello everyone, I think it's up to time, about time to start. So welcome to hear this presentation about acceptance testing with a robot framework and Selenium with Plo4Plone. My name is Ask Soka, I come from Finland and I work there for a local university

at TivaScula. So here is my contact information, so if you have a later phase problem with robot framework and Selenium and things like that, you can for example write a question at StackOverflow and then post about that question at Twitter and tag that Twitter with my Twitter handle

so I can see the question and maybe answer it later. So while I'm here it's quite an accident, I think we heard about robot framework for Plo4Plone community at first time in San Francisco 2011 at the conference there. There was a code flow chapel was present in robot framework for us as an answer to better

test our JavaScript in a Plo4Plone site, but back then the stack was not yet quite complete it was hard to integrate those tests with a normal Plo4Plone test workflow and then about half a year past then I needed that test feature to start working our

testing our Plo4Plone instance, Plo4Plone add-ons. And I just wrote the first integration for Plo4Plone and write a blog post about that and then somehow I ended up being the person that everybody asked about these things. And that's how I ended up here to tell you about robot framework and Selenium testing

for Plo4Plone. And I'm here trying to tell you that it don't need to be difficult, there are many ways to do the testing, there are easy ways and harder ways and like simple ways and more complex ways and both have cons and pros and I think I believe that one way is not the best for all, so that's why as I promised in the summary I present a few

different ways to do the testing and you can pick that one which fits best for you in your use case. Also to get into this topic here I run a few example tests from Plo4Plone.

So about the menu. So at first I present to you a few terms so we know what is accident testing and what is robot framework and what is Selenium. And then as promised I walk you through a few different ways to do this testing with Plo4Plone. So you can start only with testing your live site, of course not doing any dangerous things

for this and then slowly making it more integrated and more like the way we are doing with Plo4Plone core. And then at the very end of the presentation there is the most fun way of everything so how to write documentation so that your documentation will include embedded robot

tests so that those robot tests will generate screenshots for your documentation. So that's why you don't actually kind of write tests at all if you don't want to but you just write documentations and while you generate the documentation you kind of test your Plo4Plone add-ons and sites accidentally.

Okay so what is accident testing? It's functional testing of your product or system or accident user stories for the systems and what are user stories that are also called often as accident test criteria.

So when you're developing something for a customer, a customer have a feature request and from those feature requests you should write user stories as Dolan was telling us yesterday in his first presentation. So user stories are like concrete examples how to use those features and those user stories

for those features can form like a set of acceptance criteria for your add-on or your product and if those test pass then those access criteria are met and the product is

kind of ready for shipping and it's quite popular to write these tests in a language called jerking, it's also called BDD syntax and it's a way to write those tests as a high-level story using the real language of your customers.

So as in Plo4Plone we have editors who are doing things with Plo4Plone so we use a story where the editor is here, he is the owner and he has a document in English and then he wants to translate that English document in some other language and when he has translated

it then he sees the final documentation. And of course this actual user story or test case from product multilingual and later this test script is split into smaller and smaller parts that do the actual testing

but that way there is always a high-level description of the test that's readable for everyone and understandable for everyone on that particular domain of course. We often see things like that from the test advocates, you should do a lot of unit

testing and so unit testing are the things that test that your code is actually working correctly so there are no bugs in your individual functions or methods of your code. And then of that you should write maybe less but at least some integration tests that test that all your pieces of your code work together and work together with

the other packages. And only on top of that level you have a few acceptance tests or also called functional tests that test that the whole package works. But I think it's a matter of resources and your background, what do you like best.

So usually it's compared that okay unit tests that your code works correctly works right but only acceptance tests can test that you are actually doing the right thing, that you are doing the thing that the customer required you to do. So it's always balancing which one you want to focus on.

Of course in Perfex Pro you do all of this and of course in Plone because Plone is so complex that we have I think this green bar is maybe the biggest place because we always integrate with other packages so most of our tests are integration tests I think. And also this test level change differs on how good error messages you get when your tests fail.

Usually when you do acceptance tests they test the whole system from the very top level and it's kind of black box. You usually know that okay this feature is broken but often you don't know very well

that what is really broken there and to really get to know what is the real issue, the very technical issue then you would need unit tests and integration tests to really tell what thing is breaking there. But also if you can only integrate a unit test the real product may still be broken.

And we have seen also that with Plone many times. So I don't force you to focus on anything of this that I think you need to always think about your resources and customers. So if you have good customers who can describe your good features and also maybe participate

for writing those usual stories then acceptance tests may be the easiest way to go. And why I like acceptance tests a lot is that it's often hard to decide what should you test. So I have a bunch of code what should I test? So that is that one plus one is two or something more complex.

But from the acceptance test usually you know what you are doing I hope. So you are testing a thing you are doing from the end user perspective. So that way it's kind of easier to understand what you are testing. And only later because acceptance tests are also quite slow to run

so usually when you get better in testing then you start liking more and more integration. You need to test because those are faster and you know better how things can break when acceptance tests are breaking often. So I like starting from the acceptance test and then going slower towards integration

unit tests. Also there are some completely different opinions that unit tests are the most easiest ones to write because you only need to test the thing you have in your actual code. To continue with robot framework. Why robot framework is a good tool for writing acceptance tests?

Robot test framework is a very generic test framework. You can test anything with that. I think you can also write unit tests with that. It's mainly written for acceptance tests because the very core of the framework is a keyboard driven language syntax. And it means that you can build your own test language

so you can write your customer domain basically. And it's easy to compose those customer domain key words and make them a higher level keywords, yet higher level keywords.

And at the end you have a test case like a readable user story and then it goes split down into smaller and more exact keywords. And also that's very special for robot framework. It's very easy to do with that. I think it's much easier than writing that test in poor Python for example. And of course it's not, it's just blown on Python specifically.

You can also extend, it's written in Python so you can extend the framework easier to write in Python. It's a simple way to write Python libraries that bring your new Python key words. Which you can use to build your own custom key words. So you can write key words that use your own key words

and that way write a high level test case. But also you can extend it in Java or XML RPC services. And this is interesting if you are writing tests for like separate systems that work that you cannot kind of get direct access to the code of the system

But you can still write services on the system. So you can write your kind of testing service that you can call with robot framework. And that will test your system with hardware key words. And well, installing robot framework is really easy. It has very few dependencies. If you know Python you just create a Python virtual environment

and then install robot framework. And it installs your command line tool called Pybot and it has quite good help. So you have the basic all the options there are. Also robot framework has very good, almost too comprehensive user manuals at the internet.

But it has also sort of tutorials and quite nice slides shows how to get started with it. But there the basic uses that just use robot framework to test something. It just install it and run it. And here is very very minimal robot framework test suite.

It simply, robot framework has built in test library that's kind of built in test you can use in all test. But on top of that you can include any libraries you want. And for example in this example I simply import string library which is the basic library in Python, standard library.

And then test that that library works it can translate my string to upper case. And this way you can actually just test your Python libraries without adding anything between the robot framework and the library. Just import the library with library settings keyboard and then start testing it.

Of course if you need some more complicated in between you can write a special testing library that knows how to call your library. And robot framework test suites are simple plain text files.

So there's one file for one test suite. And there can be multiple test suite in directory or as usually. And the one test suite can contain as many test case as you want. And the test case can contain all a few different parts or settings that you import libraries and you import your other robot framework files as resources.

So you can define your custom keywords also in this plain text robot framework files and then include those fly files from into your test case as a resource. And then like have one common robot framework file you use in all your test suites. And then you can define variables using a test case.

Those variables can be filled from command line. And that's the way how we make like generic test setup setups for CEI systems. So we can like overwrite things like in which port the system is running with the variable. So we use the variable as a port in these test cases and then we overwrite that variable from the command line.

And that way we have test we can run multiple tests simultaneously so that those test use different ports and don't have conflict. And here this keyword part is the main thing in robot framework.

So yes you can build higher level keywords from lower level keywords for as many levels as you want to. And those keywords can take as many arguments as you want to. So it's kind of a broken language of its own but it's plain text and it's simple.

The hardest part in the syntax is that in robot framework can be written in a few different ways and this plain text syntax is the most popular one. And the only one problem here is that the default separator here is two or more spaces. Or you can also use tabs or pipe characters but two or more spaces is the most popular one.

And the most common mistake is forget only one space somewhere and then it goes broken. But that can be fixed by using some syntax highlighting add-on for your editors. Another nice feature of robot framework is that it creates very nice test results for you.

So very understandable. Here the tests have just few more tests broken so the whole background is red. If everything is passed the whole background is green. And it shows you nice overviews of your test and then you can go in the very detailed level to see you.

So every step in the test is locked in the results file and you can see the exact step that failed. And when you are running Selenium test as we saw later, the Selenium integration will also take the screenshot of the situation where the test failed. So you will have very detailed test results with screenshot of the fail and things.

And also this is very nice blocking for Jenkins which is the popular CI system. And easy to use and it will generate these results and show them nicely to the Jenkins UI.

And also if you have Windows users you can use this IDE for robot framework that also helps you to not make those one space mistakes because it gives you kind of table when you are writing those tests. And of course because it's Python it's a little difficult to get Python things installed in Windows.

But if you have someone to install it for you then you can just install robot framework IDE and then start testing the external system with it. So that's writing keywords there is a helper for showing all the keyword documentation easily. And then buttons to run the tests and also open those results.

All from Windows UI without needing to access the command line. But then the last part in our terminal is Selenium. And Selenium is the most popular browser testing, browser automation framework.

I almost said testing because it's mostly used in testing but it's just a way to make your browser do things automatically, programmatically. And I think it's the most popular and most well supported. Of course every now and then there comes new browser version that goes somehow broken and then you have to update your Selenium integrations for that browser also to make it work.

But usually it just works. Firefox works by default without any action. Also PhantomJS seeps the Selenium web driver integration by default with it. So those browsers work without any additional installation. But I think Opera, Safari, and Internet Explorer will need some, and also Chrome will

need some special driver to be installed that you can use those browsers in your test. Also if you don't want to install browsers for yourself you can use external services. The source lab is a SaaS service for Selenium testing. Also many other things you can use their interface to write the actual test.

But you can also run your Selenium test and Selenium robot framework test there. And just choose any browser you want or run the test suite for all the available browsers. They have all the browsers and all the platforms there. And they are free for open source projects.

And then there is a new service called WarCup. It's not compatible with robot framework. But it's like through the web system for you to test live sites. And it's made for writing documentation, new documentations, and like screencasts for live sites.

But you can also export the result as a Selenium file that you can run on your CI. But that needs a little bit more effort to do. And also, well I think there are basically two ways to do Selenium testing.

Either you use robot framework like language to write the test and then run the test. And then update the test when it starts failing or things change. Or then you use recorders. So you just click your browser and the recorder records all your clicks. And then you can rerun this recorder on your CI as much as you want.

And the basic difference is that when you use the recorder the test is usually very much harder to keep up to date. Because it's only very technical language and very exact click position and things like that in your file there. In your record file. So it's probably unmanageable.

And I think that's also a problem with WarCup. WarCup allows you to go to change every step you have recorded. But still I think maintaining those files are a lot of work. But it's an interesting service anyway. Because you can start using it through the web and it's easy.

And you can also get screenshots from there. And finally Selenium 2 library is our integration for robot framework and Selenium. So it's just a Python library written Python extension library for robot framework written in Python. And it integrates the Selenium Python bindings for robot framework.

And basically you can start using it simply by installing in your Python installation robot framework Selenium 2 library. And that also pulls the robot framework. Actually getting started in testing live instances of robot framework. You only need to install this library. And then start writing and running the test.

And if you have someone to install this for your robot framework IDE on Windows. Well it also works on Mac. But it's running on something called VX Python that's currently not too easy to install on Mac. But it works well on Windows. So that's enough to get everything working in Windows.

And this is a very simple use case for Selenium 2 library. I import the library, robot framework. And then I write my test gate. Here it opens our university website and shows that something is available on the test website. And then at the end whether the test passes or fails it should close all the browsers that opened.

Otherwise you will have open test browsers after you are running your tests. And that's not nice. So basically this is everything you need to start testing your live sites. And the easiest way to start Selenium testing I think. And I don't see nothing problem with testing live sites.

You can test everything your non-log in users do with your live sites. Of course if you get the failing test on your live sites that means you have already pushed something broken on production. But still if you see that faster than your users then it's nice to see that it's breaking.

Also some other valid use case for this would be to keep your caches warm. So run this kind of test with regularity so that they can make some complex searches or things like that on your web pages. And make sure that the most important things run fast and smoothly on your site.

And even one more where I cannot go in detail in this talk. I have written a webinar around Wave Tool Park or Accessibility Tool. So with this kind of integration only you can use that accessibility tool to actually test that your site live content is accessible.

And with little more integration you can ask for all the possible URLs you want to test. And then you just run the accessibility tool through robot framework for all those pages. And then you see which pages have more accessibility issues as your threshold.

And that's very interesting. I think I'll come back to that next year when we have more strict accessibility rules in Finland in action. And going back just for this picture so that this is everything you need to start testing.

And everything else is extra and making things more automated and more compatible with CI systems and like we do in BlownCore. Yeah. And here's Mikko with Katana.

To go a bit further is that there's a hidden gem in the ecosystem that's called GT before storage and demo storage in TEO. And if you're running your BlownCore as a TEO system, so TEO server and instances,

you can start a local instance with such configuration that it gets your live data. But it doesn't allow you to modify it. Actually you can modify the data but it modifies it locally in memory or in local storage on top of the actual live storage. This is very, very neat. Of course you could just copy your database from production to your own computer.

But if your data is multiple gigabytes or 10 gigabytes that is not so easy. Of course you have to be careful with this that you really have these settings configured and they really go into your job configuration and so that you are really not modifying on your live system.

But when you get enough trust into this configuration it's quite neat for doing depooking and also doing this test. And also you can use this to introduce a new version of your code in your system. You can test your production code with live data with this configuration.

Either on your local system or CI systems. And also the other product here is printing mail host that prevents your Blown, local Blown for sending any emails. So if you are playing with live data you usually have a live email address. You don't want to accidentally send emails from your test to your real customers.

Yeah, and that's the second level kind of testing. So you only run the local Blown instance to run your live sites but nothing too fancy. And yeah, this is the most difficult one way to do testing.

The Blown test doesn't look so difficult yet. So we have a Blown robot framework that brings in the creations for our robot framework testing. Maybe the main feature of that add-on package is the open test browser keyword. That it parameterizes opening the test so that we can override all the necessary things in CI.

So that we can run things in parallel. So that we have a common way to write tests so that we can run all those tests for different packages on the same CI in parallel. So that they don't use conflicting ports or other conflicting things.

And we can also vary the browser to run the same test for any browser we want. And also the package brings integration for source labs so that we can record the test results at source labs better.

Another way here is... So in actual test files we only need to import our Selenium resource from Blown robot framework. And then we import also something called remote library. Called robot remote. That's an XML RPC library inside Blown.

So our test fixer that is in Blown robot framework actually installs a small tool inside Blown that we can do very fast calls outside the tests. Outside actually testable things to prepare things. And I later saw why is that.

Yeah, so this is like a minimal test case. But to make this test actually run with our normal blown up testing sandbox, it's a little bit work. So we need to... At first we need to wrap that test around our blown up testing... Blown testing runs with Python unit test. Test switch. So we need to wrap that robot framework test into Python unit test.

Test switch. And that's the first integrating package I create the robot suite for that. So there is one more file we need to file. Usually it's called test underscore robot.py in product test directory that has this wrapper around it.

And it just makes... returns a test suite of all the robot files to test. And then it uses a blown up testing functional testing fixer. Which also includes the remote bundle to enable other things faster.

And finally we need to include the whole robot framework resources in our advanced test file. But this more complex, more involved advanced testing is quite well documented as a tutorial.

And the blown docs. So just go to docs.org and search for robot framework. You should get a few results with at least one is the correct one for the tutorial. Start from the writing blown up robot framework testing for your own add-on. And how is this now different from testing the actual sites as I saw earlier?

This way we can run tests for multiple products in one test run. It's like we have... blown has hundreds of packages. And then those packages have robot tests. And this way we can run all robot tests from all those packages in one test run with all the other tests. So it integrates into blown core test flow and also maybe your own own CI flows.

Yeah, so earlier I said that we often write tests in BDD syntax called shirking. Well, that's not necessary but it's a nice way to help you to structure your test.

So BDD syntax splits your test into something called given clauses. So given something and something and something and something. And those are something that are not the one you are testing. They're like preconditions for your... it describes your test environment with the condition and the state of the system you are testing.

And only then you have something called when clauses. So when I do something and I do something else and I do yet something else. Those are the actual things you are testing, that you are doing. And then at the very end you have those assertions that after I have done these things, then my system should be in some particular state.

So I assert, okay, when I have... at the beginning I have a document in English and then I do some things to translate from English to Catalan. Then at the very end I really should have that document in Catalan. And now previously I saw that robot framework, robot remote, remote library.

And that's used for us to speed up the test for in these given clauses. So when we are just preparing the state, we are setting a user and logging a user in and creating a document in English. We don't want to click these things to send them to browser because that's slow.

But through the robot remote library, that XMLRPC library for Plone, our robot framework, we can simply create all these on the background. So when we do the first when step or I open the document in English, the document actually is already there and I only need to click the translate button,

translate it in English. And so I've saved a minute of time at least in the test. And that really counts up when you have a lot of tests. Of course when you are starting and learning things, you can just write. And this is only problem when you have a lot of tests and you are getting more professional.

Well, this is extra. You don't need to remember this. I've already uploaded this slide for Slideshare, you can find it. But previously, I thought that you can add a local Plone instance

that have a local add-ons and can have a local development version of your product. You can actually also add into that build out this robot remote library so that you can run those tests using also that remote, robot remote library

and call those keywords to log user in automatically and create the stuff background to make also those tests faster. And then the last part, there's still some time left. So the most fun part is the latest integration we did a couple of years ago.

I think last year, summer, at the Google Summer Code Project, we created things integration that means you can write robot framework tests inside your things documentation. So it's simple, you use the standard restructures that code directive,

give the language a robot framework, and then you can just write your robot framework code into there as usual. And when you run your Sphinx build, it will actually run those tests during the build. And by default, it will also solve those test results as soon as highlighted code in the documentation. But there's an option that you can make it all silent.

So this is kind of new way to do doc tests. Doc tests are the way to try documentation for Python libraries so that you have tests inside the documentation. And they have very bad reputation because usually we ended up writing two technical tests there and they don't really tell you how to use the product.

So the document testers came more like how you test this product and how you use this product. But maybe with robot framework you can use such high-level keywords here that the test code even could be more descriptive as to what is really happening. But usually I just hide this and use the next thing that is

I use those tests to create screenshots about the product I'm testing while I'm writing things build out. And yeah, this is a little more involved. So we have special keywords in robot framework that can, when running the Sphinx build out,

actually start the blown testing sandbox and then we can run normal blown test within the Sphinx compilation, document compilation thing. There's something new from the previous slide. There is something called server resource that brings the keyword for set up in our blown site with our testing fixture.

This is the default blown server fixture with only default blown. It can be also your own fixture. There are examples for example blown that multilingual has to set up. And we are adding more of these into blown core documentation so that we can create the document screenshot for blown documentation with this approach

as Eric said in his keynote. And then we take the screenshot. Those are not the library code, it's element two screenshots. It's made by me. And basically it gives you a few keywords for annotating the screenshot. Before taking the screenshot, it uses JavaScript to add things there. And then the main thing is that it can crop the screenshot using the CSS selectors.

So here I'm going to blown home page and then I'm taking a screenshot of the portal footer. So I want the screenshot to be cropped for the portal footer. And then I tear down the blown sandbox and close the browser.

And at the very top there's a figure directory which is the normal things, figure include directory. So when you run these things, it first says that okay, that portal footer B and C is not there. But then it runs the remote framework code and then suddenly it is there and it will include in the resulting documentation.

And here's for example, example run for documentation, current documentation of blown at Mosaic. It takes a little time to start the sandbox. Then it clicks to the site and tests things and takes the screenshot while doing it.

And also this, by default there's things running, when we run this in things builder, we don't generate the test, test results reports, we only build documentation. But also these things documentation can be run with the default PyBot, the default robot framework test runner.

So we can, we have both the screenshot built and then we can have the real test built with real test results. So we can use the same set of files for both getting documentation and creating the test, the test for the product. And yeah, there's one downside of this approach is that

previously I showed you the way of writing the test. And that allows you to write understandable and clean test, maybe even beautiful test suits that are easy to keep up to date because it is to understand what they are doing. But with this approach the tests are, when we compare here,

this is more imperative. Making these test case use PDD would make the documentation files very, very long. So usually when I do this I just use the right keyword and then the test case code is a little bit spaghetti afterwards.

I didn't plan to do that so let's go back. Skip that, yeah. And this is the result in documentation for Plone Mosaic currently. So there's the crop screenshot of the things we saw previously.

There's even tracking going on for one Mosaic tile and then there's descriptive text around it, like in normal documentation. And that's cool. So I think we have more than five minutes left.

Here are the main links about this talk. So the robot framework has a very nice front page. It gives you the basic slides and basic interaction for robot framework and it also lists all the possible libraries for it. So you don't need to only test Plone with this and you can use it for any products. You can test many kinds of systems with it.

It has SSH library and almost anything you can even think of it. One very interesting is called CQL library which uses to test with CQL framework. CQL framework is a framework for comparing images. So it's more like testing for desktop add-ons but you can actually also test does your CSS go broken.

So the screens are very different and then it raises an error for that. And then there's a keyboard for Selenium 2 library. That's the main library we work with robot framework with Plone and it tells all the way how to click the browser to do the things and test things there. And then at the end is my blog post from last year

where I describe all the add-ons we are using in Plone testing cases. Yeah, and we have a few minutes for questions.

So one of the things that I've been trying to work on teaching some of the students that I have for other Python web frameworks is to use a BDD approach to developing websites. It looks like this would be an excellent way to use that on a Plone site. Do we have already keywords that tie into some of the underlying systems

like the email system so you could test content rules that you've created and ensure that when something gets published an email got sent to the person responsible for reviewing that or some? We have a very basic key for deep and hidden in Plone robot framework.

There's a test fixture that installs, actually the robot framework remote bundle I think it's one test fixture that needs to be included in robot framework remote bundle in one keyword that gives you the latest email sent with only the plain text. Very basic, it can be enhanced a lot. Can you please recommend a GitHub package which uses Selenium 2.0 library

or the robot framework working tests, just an example?

For Plone I can recommend Plone with multilingual. It doesn't have so much test but they are good quality. We designed them last year in some sprint particularly for good examples. So, Plone app multilingual and they are browsed through the Plone slash app slash multilingual slash tests and slash robot.

Well, thank you.