It Takes a WebKit to Test a WebKit

When I left off last week I was going through the trials and tribulations to build WebKit in a Linux OS environment. It was a long week of desk-banging and face-palming, but eventually I managed to drag my way through it. While my original gut feeling after that event was that I would really not like to see any more of WebKit, there is more that needs to be done. The purpose of building WebKit was not to simply see whether it could be done (which would be completely useless to our implementation of WebVTT), but so that we could gain access to WebKits tests and test harness in order to see the kind of testing that WebKit did for their implementation of WebVTT. Now that a working build has been created the next step to that end is to learn how to use WebVTTs test harness.

The obvious first place to start would be the information webpage for the test suite in WebKitGTK which can be found here. This page describes the call method of the run-webkit-tests function which starts their test harness. From this page the command to run all the tests in the WebKit test suite would be:

Tools/Scripts/run-webkit-tests --gtk --debug

This starts the test harness which checks the validity of ~27000 files. While theoretically testing like this should be quick, there was a default timeout timer of up to 30 seconds! Unfortunately during my initial run of their tests 200 files timed out in this manner, and thus it was important to find a way to test only the tests pertaining to WebVTT. This can be done with the addition of the folder path to the command, thus since we know the folder path of WebKit’s WebVTT tests the modified command is:

Tools/Scripts/run-webkit-tests --gtk --debug media/track

Which launches a much more manageable 71 tests.

From this I found 3 major interesting things:

1) Not all their tests pass their own test harness. 1 of them fails on a text diff, 20 of them fail on timeout. Further study will need to be done to understand why this happens.

2) Their test harness uses the .html files of the tests, it does not test the .vtt and .txt files separately. This means that if we want to run our tests through their test harness our tests will need to be converted into the same format .html that they are using.

3) Their test harness automatically retests any tests that fail initially to ensure that the failure was not an isolated incident. Not absolutely essential to us, but interesting to see.

When the testing is completed their test harness shows the tests that passed and failed on the terminal window:

and also a more detailed description of the failed tests in a barebones browser:

which very helpfully includes a diff of the expected and actual results:

This is where I left off for the weekend. In the future I will be looking into the reason those 21 tests failed, as well as figuring out how to convert our tests into tests that will work with their test harness so that we can check how their implementation holds up.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s