My name is Doug Gorman and I'm the QA manager for Creative Artists Agency. We've adopted Umbraco as our default standard for business vertical sites and my team is going to be validating these sites fairly quickly. We've basically made the decision not to focus too much on testing Umbraco's capabilities but plan to instead focus on our customizations of Umbraco.
The question I have for other development or QA teams who've implemented an Umbraco site - what was your test plan? What strategies helped you find the most bugs prior to launch with your implementation of Umbraco? Did you put much effort into testing the CMS or did you focus on the templates you applied to it? Were certain technologies like Flash or Silverlight problematic?
We don't test every aspect of Umbraco functionality with each install, but do stick to only stable releases. We do test our implementation and basic functionality to ensure pages are displaying properly, templates are cross browser compatible, our macros are functioning properly.
We try to do a complete walkthrough of the site before launch, testing each part of our implementation. This is done twice, once internally, and once with the client.
We run three fairly identical installations of umbraco which house all of our sites. The first is for shared development, second is for testing and the third is our production. Each custom control or code must follow the process of being reviewed by another developer on the development server and verified on the test server before ever touching the production server.
Doesn't always run smoothly, but keeps us somewhat sane.
I am not a quality manager, but I was asked by the QA team to implement several QA processes.
We are running several servers as part of a QA flow; a production server and a failover server pointing to the same database server, a dev server, a DRP server and a test/qualification server. Files are deployed to all these server with an in house deployment tool that copies all the files from a DSL to the server, sets the IIS website and permissions from a permission template. Media files are synced across the failover and production with xcopy scripts.
Our testing mainly focuses on the templates, because we only recently changed and now we use our own compiled code for Umbraco. Before we used the binary distribution and then you cannot do much ...
For testing the template:
Editors have access to the test server via terminal server which opens a browser with a fixed proxy (localhost) to allow them to browse all the websites that are running from our Umbraco installation. This is one way of manual testing.
I am building selenium tests with an XSLT script to automate tests. It is not yet fully functional, but it works already for several document types.
I have one xslt script that creates the suite of selenium tests. Each test is attached to a document type, and additionally you could have different tests for document-types with different templates.
After that you have define your tests per document type/ per template. Once you have defined your tests, you can create an XSLT script that outputs the selenium HTML format to run these tests on a subset or all of your pages.
I was surprised to find that many pages caused errors, although most of the pages and the tests are automated.
This is the suite, for each document type find all pages and create a link to a file:
QA Strategy For Umbraco Sites?
My name is Doug Gorman and I'm the QA manager for Creative Artists Agency. We've adopted Umbraco as our default standard for business vertical sites and my team is going to be validating these sites fairly quickly. We've basically made the decision not to focus too much on testing Umbraco's capabilities but plan to instead focus on our customizations of Umbraco.
The question I have for other development or QA teams who've implemented an Umbraco site - what was your test plan? What strategies helped you find the most bugs prior to launch with your implementation of Umbraco? Did you put much effort into testing the CMS or did you focus on the templates you applied to it? Were certain technologies like Flash or Silverlight problematic?
We don't test every aspect of Umbraco functionality with each install, but do stick to only stable releases. We do test our implementation and basic functionality to ensure pages are displaying properly, templates are cross browser compatible, our macros are functioning properly.
We try to do a complete walkthrough of the site before launch, testing each part of our implementation. This is done twice, once internally, and once with the client.
We run three fairly identical installations of umbraco which house all of our sites. The first is for shared development, second is for testing and the third is our production. Each custom control or code must follow the process of being reviewed by another developer on the development server and verified on the test server before ever touching the production server.
Doesn't always run smoothly, but keeps us somewhat sane.
We've not had any problems with Flash.
-Chris
Hi Doug,
I am not a quality manager, but I was asked by the QA team to implement several QA processes.
We are running several servers as part of a QA flow; a production server and a failover server pointing to the same database server, a dev server, a DRP server and a test/qualification server. Files are deployed to all these server with an in house deployment tool that copies all the files from a DSL to the server, sets the IIS website and permissions from a permission template. Media files are synced across the failover and production with xcopy scripts.
Our testing mainly focuses on the templates, because we only recently changed and now we use our own compiled code for Umbraco. Before we used the binary distribution and then you cannot do much ...
For testing the template:
Editors have access to the test server via terminal server which opens a browser with a fixed proxy (localhost) to allow them to browse all the websites that are running from our Umbraco installation. This is one way of manual testing.
I am building selenium tests with an XSLT script to automate tests. It is not yet fully functional, but it works already for several document types.
I have one xslt script that creates the suite of selenium tests. Each test is attached to a document type, and additionally you could have different tests for document-types with different templates.
After that you have define your tests per document type/ per template. Once you have defined your tests, you can create an XSLT script that outputs the selenium HTML format to run these tests on a subset or all of your pages.
I was surprised to find that many pages caused errors, although most of the pages and the tests are automated.
This is the suite, for each document type find all pages and create a link to a file:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <meta content="text/html; charset=UTF-8" http-equiv="content-type" /> <title>Test Suite</title> </head> <body> <table id="suiteTable" cellpadding="1" cellspacing="1" border="1" class="selenium"><tbody> <tr><td><b>Test Suite</b></td></tr> <tr><td><b>Document type testing suite</b></td></tr> <tr><td><a href="http://localhost/selenium.aspx?case=website">v3 Multi language website</a></td></tr> <tr><td><a href="http://localhost/selenium.aspx?case=HomePage_v3">v3 Homepage</a></td></tr> <tr><td><a href="http://localhost/selenium.aspx?case=NodePAge_v3">v3 Folder</a></td></tr> </tbody></table> </body> </html>And one of the selenium tests:
<tr> <td>beginJsErrorChecker</td> <td></td> <td></td> </tr> <tr> <td>open</td> <td>http://www.eurofins.de/unser-angebot/umwelt.aspx</td>; <td></td> </tr> <tr> <td>endJsErrorChecker</td> <td></td> <td></td> </tr> <tr> <td>verifyTextPresent</td> <td>Willkommen bei Eurofins Umwelt</td> <td></td> </tr> <tr> <td>verifyTitle</td> <td>Willkommen bei Eurofins Umwelt</td> <td></td> </tr> <!--Verify the description: property metaWebsitedescription--> <tr> <td>verifyTextPresent</td> <td></td> <td></td> </tr> <!--Verify if the logo is available--> <tr> <td>open</td> <td>http://www.eurofins.de/media/423827/eurofins-umwelt-logo.png</td>; <td></td> </tr> <tr> <td>verifyTextNotPresent</td> <td>Page not found</td> <td></td> </tr>is working on a reply...