Release testing system | Help needed

Projects
#1

I have created a website today allowing us to have a library of tests (user oriented) and - as testers - claim tests, set them as passing or failing, and have a log of releases > tests > results

Once the test cases are created we can re-use them for each release anew while keeping the log of previous test results (this will all be automatic once we’ve the test cases, we just will need to publish a new RC and that’ll prepare all test cases for actual testing on this new release)

It’s semi public, means the results can be seen by anyone (inclusive when and who tested last what, in what release).
Only registered users (id like to add forum login plugin, but for now I’ll set up users manually) can claim and test or suggest new test cases

After each Release we’ll for sure have couple more edge cases as well to add

The releases show a % of testing done, and a simple color code shows what tests pass, which fail and what’s free to claim.

Prior to each release, the release should of course have 100% covered passing tests.

When a test is marked as failing, the user is compelled to create a git issue and link it to the case.

Now, I’m looking for volunteers to start creating test cases
The work consists in adding a discernible name for the test case, a bullet list of steps to perform, and click submit.
It’s all done in the front end.
Of need - screenshots could be added - but generally it should be something like:

Title: Add New Theme
Content:

  • go to appearance > ttheme
  • Click on add new theme
  • Upload a theme
  • Click on activate theme
    Etc

Please inbox me pm if interested so I can share your user logins or add a note here „me“ and I’ll add you + send login details in pm.

Note: the system is still wip but we can start adding test cases, this is the biggest work to be done.

NOTE:
I’ll need your email to create your user. So please inbox me that mail (better than public)

2 Likes
#2

You might find some helpful stuff on the WP Test blog. It seems that they must have a place that they are putting their test step lists.

This article too: State of e2e testing in WordPress Core

1 Like
#3

I have now tested with something similar to selenium.

You set the tool up by actually recording the steps once.
Then, when testing, you press “play macro” and it goes thru those steps. If it fails, it throws an error.

Example (note, this JSON does not need to be written, it is generated for you when you click around on the screen):

{
  "Name": "Themes",
  "CreationDate": "2022-4-6",
  "Commands": [
    {
      "Command": "open",
      "Target": "https://classicpresspre.mamp/wp-admin/themes.php",
      "Value": "",
      "Description": ""
    },
    {
      "Command": "click",
      "Target": "id=classicpress-twentyseventeen-action",
      "Value": "",
      "Targets": [
        "id=classicpress-twentyseventeen-action",
        "xpath=//*[@id=\"classicpress-twentyseventeen-action\"]",
        "xpath=//span[@id='classicpress-twentyseventeen-action']",
        "xpath=//div[2]/span",
        "css=#classicpress-twentyseventeen-action"
      ],
      "Description": ""
    },
    {
      "Command": "click",
      "Target": "linkText=Delete",
      "Value": "",
      "Targets": [
        "linkText=Delete",
        "xpath=//*[@id=\"wpbody-content\"]/div[4]/div[2]/div/div[2]/div[3]/a",
        "xpath=//div[2]/div[3]/a",
        "css=#wpbody-content > div.wrap > div.theme-overlay > div > div.theme-wrap.wp-clearfix > div.theme-actions > a"
      ],
      "Description": ""
    },
    {
      "Command": "click",
      "Target": "xpath=//*[@id=\"wpbody-content\"]/div[4]/div[2]/div/div[2]/div/button[3]",
      "Value": "",
      "Targets": [
        "xpath=//*[@id=\"wpbody-content\"]/div[4]/div[2]/div/div[2]/div/button[3]",
        "xpath=//button[3]",
        "css=#wpbody-content > div.wrap > div.theme-overlay > div > div.theme-wrap.wp-clearfix > div.theme-header > button.close.dashicons.dashicons-no"
      ],
      "Description": ""
    }
  ]
}

This will simply delete Theme 2017 from the themes in CP Install.

In CP 1.3.x this will run without error, as well in 1.4.1
In 1.4.0 the error will be thrown at step 3 "Command": "click","Target": "linkText=Delete", because there will be no such clickable.
The error will be [error] Line 3: timeout reached when looking for element 'linkText=Delete', [info]Macro failed (Runtime 12.80s)

This was setup using UI.Vision RPA 6.3.3
I think we can also export those test cases, since it allows me to import some, but I did not yet find out how.

This could automate a bunch of “UX” test cases massively. And all it needs is someone to go thru the steps once, and never again.
I could for example upload the “Macros” to the test site or even create posts there based on the macros so anyone who wants, could download the Chrome thingy and import the test case(s) and start testing.

1 Like
#4

I have tested a number of tools and above is so far the best, easiest to understand and most reliable.

Result of current test suite for CP:

[report]
Test Suite name: CPQC
Start Time: Wed Apr 06 2022 18:11:48 GMT+0700 (Indochina Time)
Overall status: OK, Runtime: 63.30s
Macro run: 3
Success: 3
Failure: 0
Macro executed:
Add And Delete A New Plugin  (OK, Runtime: 20.30s)
Add And Delete A New Theme  (OK, Runtime: 17.09s)
Add And Delete A New User  (OK, Runtime: 25.63s)

If the tester is smart they also enable debug log while running these tests.
That will then also show if some dangling PHP notices, errors, or else are thrown.

Gosh, this is actually cool

We won’t have to write stupid lists of “do this/do that” for most cases. The tool is free and available for Mac, linux and windows since mostly it is a browser based add-on. For team sharing (thus local storage of the macros) an app must be installed, but again it is free for all 3 major platforms and no data is sent to the cloud.

It is fully selenium compatible and thus also uses their commands.

I have a feeling that sooner or later I will run in to some “aha” limitation. Such as the test numbers one can save locally (10 only). So either make big enough tests (not ideal) or buy a lifetime license.
Otherwise an idea could be to share the browser localstorage, since that can be copied and added by any user to his browser, and then there are no limitations…

#5

I see your JSON has your https://classicpresspre.mamp/ in there. Will this be a problem?

#6

What is selenium, besides an essential mineral?

#7

No, because this is just an example.
In real world, there is a website to test with and that url will be used.
OR, we can use a variable, that gets set in the tool and then the tester sets that variable in the settings, to work for example with his local site.

Selenium is the industry’s standard for such tests. That’s what I referred to in #core channel when I mentioned „we can do browser testing, but I don’t know how to implement it“

The reason I didn’t use selenium itself was I … thought it’s a paid tool. And now I can’t see their „pricing“ menu anymore, so I guess I had looked at the wrong site initially and I’ll test today with selenium itself - why use a hybrid if we can use the real thing…
Will see how it goes

1 Like
#8

Tested the selenium IDE - basically the same as UI.Vision, and their ecosystem is more robust (scaling, etc)… but, I couldn’t find the equivalent to easily assert visual content as I can do in UI.Vision
That is very useful for example to assert that the last bug we had (missing line breaks in tiny) does not happen. I can assertVisual the exact content of a screen pixel by pixel and thus also line breaks using one single simple macro step based on a screenshot.
In Selenium that seems not possible, A) they do not have such command and B) I guess it would be possible somehow, but probably not with such easy steps.

Additionally I did not like Selenium Support Forum (google group :face_vomiting:) versus UI.Vision using the same forum ware we have here at CP. As well some components seem to require Java knowledge - nothing I would want to deal with.
The pro point for Selenium seems so far I have no limits at all, in terms of storage and disk storage, and it clearly has more powerful add-on tools (free as it seems) to scale up, and is (at least some of it) recommended by W3C, making it de facto the only tool for automated testing recommend by them, and thus browsers will be basically “guaranteed” working with the Selenium tools.

Bottom line I think while (probably by much) more powerful and scalable - selenium has a too steep learning curve, and does not “feel” right to me, while UI.Vision was “uncomplicated and cool” from the very first moment I used it, and has features I did not see on Selenium such as Computer vision assertion, which is very, very helpful.

I think for the purpose we would use it is ok - the only drawback is, using their “cool” methods like visual assert means it will only work with their tool, while selenium would be compatible with anything out there, and the cap on disk storage is uncool.
But as said, it requires much more time in selenium to actually do the same what I did in just a couple hours with UI.Vision (if even possible), and while it would be cool to get certified for private work purposes, this is one of these situations where I think “get things done now is better than getting things done in the most extremely professional way, but probably never”

1 Like
#9

I vote for visualUI. I mean, if it is simple, scalable in the future eventually, gets the job done… It’s the better tool for now.

#10

Were you planning on visual tests only, or tests that involve the database or files (uploads, update, delete)?
Perhaps visual testing is one tool and the other tests are another tool.

#11

These Tests are capable of both things.

#12

As a sidenote, relevant to testing, ClassicPress has an open source plan subscription to BrowserStack.

1 Like
#13

Thanks @viktor - I did not know we have that. Looked into it now, not too long thou.

From what I gathered, it seems this tool/service is
A) Very tool specific (you seem to need for each and every type of device a test)
B) it is Code only, as far I saw

It is certainly much more powerful, yet, with our resources, no real code expert involved in this, and the sheer impossibility to keep such tests updated (not to mention to run them if you do not have either tech knowledge OR we create a full blown automated test suite run thru CLI), that is not suitable for this sort of testing.

I think it is not surprising that while we seem to have this service since 2 releases, we have not used it. Sad, I think - but it’s just as it is.

Now, I am not trying to create a test suite for testing CP on iPhone 6x or Android (xy) or similar “device/browser” related testing.
I am trying to create a test suite to test ClassicPress features both with automated and manual or mixed tests, have an easy interface for less tech folks to help testing, and keeping a log of that, while having most as possible automated.

My rule of thumb is approaching this:

  • if I do not understand it, I cannot expect a user to understand what’s to do.
  • if I do not understand it, someone else needs to do it, or I have to learn it.

We have no one interested here in doing it (correct me if wrong) and I have no will to learn it (I already chose UI.Vision over pure selenium because I do not want to learn yet another coding paradigm), write thousands of lines of code, for something that - being fully honest - only very few will care for or use.

Of course, this is something that if explored (the tool linked by you) is certainly super professional and powerful.

Trying really means doing :smiley:
here is the site https://cpqc.vps.webdock.io
logged in users only can see or interact with most of the details, as explained in the opening post.

#14

How will you handle the various configurations for your testing? I mean single site, multisite, installed in subfolder, moved content folder, low disk space, logged in, logged out, different capability levels, no JS, PHP 8.1.
Or does any of that matter?

#15

If someone has the time to write all those sub tests (if then they are really so different) we can have all of them just like “normal” tests

However for now for me it doesn’t matter. I assume a vanilla cp install, not a customized or special install.

Some testing is better than zero testing, and it’s extendable of course, since it’s just a directory with some handles to claim or pass a test (plus the tests to download if automated or steps to perform if any…)

For example you could write a test to (manually if you want or automatic whatever you prefer) create a multisite.
Happy to show you

Ps:
Ui.vision itself does test whatever test you run on what you install
So I’d say you install php 5 cp if you want to see if it works, as example. Tests will be like users, so it shouldn’t be different on php versions. Same with multisite. You can set urls to test OR if specific multisite tests like setting such system up need to be done, these tests have to be “written” and then performed

No test in the world van catch ALL possible environments or setups in ONE test. They are all specific to SOME case. So for example if we know an issue happened in say Firefox but not safari, we’d have to tell the tool to test that, specifically.

#16

I saw this and thought it might help something.