What is a slow test question?

Introduction

“Slow test questions” that go around forever in test automation. There is no one-size-fits-all solution, but you have to devise a little bit to reduce the execution time of automated tests. Using a GUI-less browser will help reduce slow test problems.

test question

In the first half, we will explain slow test problems and GUI-less browsers, and in the second half, we will use headless mode.

What is a slow test question?

First of all, what is a slow test problem? Test automation takes longer than expected to execute, which affects software development work.
You might be wondering, “Hmm? If you automate the test, the implementation time should be shorter than the manual test?”
Of course, it is. “Slow” here does not mean that it takes time to execute each test, but that it increases the execution time of the entire test that is executed at once.

Even if one test execution is completed in 1 second, it will take more than 1 hour if the number of tests is 5000. For example, if you use a mechanism such as CI to execute all of these tests for each commit, you will be able to run CI normally only once every 1.5 hours.

With this, the special mechanism is the treasure rot.

Slow test questions are a “necessary evil”

Of course, test automation should be promoted. Manual testing can take many or even tens of times longer. If the test automation efforts themselves are working well, slow test problems are like a “necessary evil” that will inevitably occur.

In other words, the more efficient test automation is, the more demands you have to run a lot of tests, but on the other hand, the time is finite, so if you keep doing the same thing, the number of tests that can be run will surely reach a plateau. That’s why.
Although it is a “cat and mouse” that must be constantly improved, the occurrence of slow test problems should be regarded as improving the efficiency of the development process.

How to deal with slow test problems

There are several ways to mitigate the slow test problem. For the column, I will omit detailed explanations, but if you divide it roughly,

• Increase the number of tests that can be performed per hour
• Reduce the number of tests to the required and sufficient number
• Reduce the execution time per case

There is such a thing.

To increase the number of tests that can be executed, you can simply increase resources such as hardware.
On the contrary, there is also the idea of ​​reducing the number of unnecessary tests by performing only the necessary tests. There may be workarounds such as understanding code and object dependencies and performing only the tests that affect them.
And about the method of reducing the execution time per last one, it is especially effective in the E2E test (end-to-end test) where it takes a lot of time to execute one test. To do.

Web system E2E testing and execution time

The E2E test is a test in which the system is assembled and operated consistently from the user’s operation to the internal operation of the system, that is, from the operation side end to the system side end of the system.
Since it is tested throughout the system, it takes a lot of time to execute each case compared to testing a single module. Especially in the case of a Web system, the time it takes to start and execute the browser operated by the user is extremely affected, not to mention the operation on the system side.

I think that everyone is reading this column through a web browser, but it may take some time for the home page to be displayed when the browser is started. Recent browsers are getting faster and less uncomfortable.
Even so, the screen is not displayed immediately after the browser is started. It has almost no effect on human operation, but when it is executed hundreds or thousands of times at a time like an automated test, that little “waiting” is a big overhead (extra processing). ).

The reason why it takes so long is that in the case of a GUI application such as a browser, not only the data processing but also the processing of rendering that information as screen information and displaying it on the screen by interacting with the OS. Because you need to do.

What is a GUI-less browser?

If there is a problem like the above in the automatic test using the browser, “Then, it is okay to eliminate the screen display”, so what is called “GUIless browser” or “headless browser” has been developed. increase. It is a headless browser from “eliminate ( less )” ” overhead “.

Previously, there was “PhantomJS”, which was often used as a typical GUI-less browser. However, development has recently ended and the repository on GitHub was archived in June 2018.
Instead, Chrome and Firefox, the web browsers currently used by many users, can run in headless mode.
Instead, rather than implementing those features, PhantomJS is no longer in development.

When testing web systems, compatibility between browsers is a constant issue. Browser compatibility is also a major issue for GUI-less browsers.
The reason why PhantomJS has been used as a typical GUI-less browser is that it uses “WebKit”, which is the rendering engine used in Chrome and Firefox, internally.
In simple terms, it will look the same as those browsers.

However, these browsers now use different rendering engines, and as browsers become more sophisticated, the differences in Javascript execution engines have had a major impact.
In addition to the high functionality and diversification of such Web browsers, the spread of E2E tests due to the improvement of test automation technology, it may be said that it was a request of the times for official browsers to support headless mode. Hmm.

GUI-less execution

So far, we’ve talked about the “slow test questions” that come with test automation and the “GUIless browser.” From here, let’s try out the headless modes of Chrome and Firefox and see the difference between with and without headless modes.

The operating environment of the author is as follows.

• OS: Windows8.1 64bit
• Execution environment: Python3.6.4 64bit
• Browser used: Firefox, Chrome
• Browser operation library: Selenium web driver

Of course, you need to be connected to the internet to set up and run.

We will not go into detail about the environment construction but only give an overview. It doesn’t mean that you do complicated things.
First, install Python from the official website.
After that, execute the following command at the command prompt to download and install the Selenium web driver for Python from the Internet.

pip install selenium

Obtain the driver to operate the browser together with the installation of Selenium. Have a Gecko Driver
for Firefox and a Chrome Driver for Chrome. Place them in the same directory as the source code files.

Now, let’s write the program. Name the file with the code below “headless.py” and save it in a directory of your choice.

Run the file with this source code in Python. Try executing the following two commands on the command prompt.

cd [path of the directory where you saved the source code]
python headless.py

This will save a screenshot of Google. At that time, the screen is not displayed.
7th line of source code,

    options.add_argument ('--headless')

Please add “#” to the comment out at the beginning of the line.

    # options.add_argument ('--headless')

Then Firefox will start up and do the same thing.

Verification of effect

Run it several times and measure the time it takes for the program to finish. In my environment, headless mode runs faster for about 3-4 seconds.
To see how much difference there is actually, I ran it 20 times in different modes in Firefox and Chrome and took the average.

The headless mode was about 2.5 seconds faster, with a percentage of 15% to 20% faster. In this code, the only operation after opening is a screen capture, so it is not a big difference, but if you run a lot of test cases, it will make a bigger difference. I will. However, it does not change by a factor of two or three, so it should be considered in combination with other methods to solve the slow test problem.

summary

The headless mode has been added to commonly used browsers, making it easier to automate E2E testing using a web browser. However, slow test problems are not completely resolved and will require continuous improvement, including improving test case efficiency. How about introducing it as one of the improvement methods?

If you ever want to know about similar things, check out the Facebook page Maga Techs.