Unit Testing CSS - Looking for a Solution
I think it’s an epic failure of web standards that CSS is the only essentially untestable technology invented in last decade – Tomasz
Talking today on Twitter with Tomasz got me thinking again about one of those problems that I come back to once in a while. Unit testing CSS. CSS development is a pain, even with some sort of system. Admit it. I actually like CSS most of the time but it’s still painful at times. Hopefully with that out of the way you feel better.
unit testing is a method of testing that verifies the individual units of source code are working properly. A unit is the smallest testable part of an application.
All testing past simple validation of CSS seems to be done visually at present.
Thinking about this from the point of view of CSS seems straight forward, but turns out not to be so for a variety of reasons. The problem lies in the cascading and compounding nature of the beast. Each individual CSS rule might do something which is self contained, but the chances on a real site are probably slim. For instance:
body {
font-size: 100%;
}
p {
font-size: 2em;
}
What is the size of the font size of a paragraph? It turns out it depends. Not just on more than one unit of source code (we have two rules here) but also on things like the browser. And how do we get this font size from a browser in the first place? I generally dislike Selenium but does it provide a mechanism for getting at the calculated DOM attribute values? Do we have to interface directly with a browser at a lower level?
wxMozilla, wxWebKit or maybe pywebkitgtz might prove useful, but I’m not sure at what level they operate. What I’m imagining here is maybe something like (excuse the Python, hopefully you get the idea):
def test_text_size_is_12px(self):
response = fetch_with_browser('http://www.google.com')
self.assertEquals(12, response.search("p").fontSize
So we could use CSS selectors (ie. p) to find elements and then assert various DOM properties (ie. fontSize) are equal to values we specify. The magic is in getting access to those calculated DOM attribute values from an actual browser engine.
Another approach would seem to be looking at visual rendering and comparing against a known good version. This seems to be something that the Mozilla folks got up to a while back to test different browser versions. Their are a few tools that might help us out here too; BrowserCam provides a paid for service, Webkit2png is a handy command line script I’ve had fun with in the past and IECapt appears to be a similar beast for Internet Explorer. CutyCapt is another cross platform webkit based utility. I can see a few gotchas lurking here. Animations or slow loading javascript would obviously throw this into disarray. But disabling these in the browser might get up somewhere. How to compare images produced I’m not yet sure, but I reasons someone reading this might have a good idea?
As the title would suggest this post does not contain the answer, only a few useful links and two possible approaches to the problem. The questions at this stage are:
- Does any of this exist already? If so who do we need to cuddle up to to get access to it?
- Are any of the technical hurdles to either of the approaches mentioned above insurmountable? If not what is the best solution?
- Does anyone except me and Tomasz even want this?
I reason their are a fair few things that would be needed to make this first practical and later standard; nice APIs, run times in various languages, and working out whether or not it actually helps CSS development to name but a few. But right now I’d go for a limited proof of concept that works on my machine. If anyone has any links to thinks that might be good starting points please let me know. Other ideas welcome as well.
One last thing; Mozilla’s latest employees are looking at the whole spectrum of developer tools. I’d love for them to start with something like this.
Comments
Hixie built a screenshot comparison testing suite for Opera five years ago: http://ln.hixie.ch/?start=1064849326&count=1
I think I remember asking Chris Wilson about this a while ago and him saying that the IE team had an enormous screenshot-based test suite.
Unfortunately those are both closed source – I don’t know if the Mozilla or Webkit teams have a similar tool.
Simon Willison - 13th October 2008
I suppose how you would go about this largely depends on what you expect could change over time – could a next browser version change the default CSS slightly? Or might it be an addition to your own CSS that caused a discrepancy you didn’t predict? Are you testing in browsers whose standards compliance is sub-optimal? Are you testing for the relative size of, say, h1 and h2 elements, regardless of how large exactly they are in absolute terms?
Certainly jQuery, for one, will give you easy access to calculated font-size in absolute measurements (pixels). I suppose you could use FireWatir, or whatever the Python equivalent might be, to access Firefox via jssh so you can retrieve values directly.
I don’t think I’ve ever found myself wishing for some obvious means of unit testing CSS, though I’d use a tool that could analyse new CSS rules as they were being typed to a file to see whether they caused changes in calculated values in other rules further down the cascade, and report which elements might be affected.
Douglas Greenshields - 14th October 2008
Getting the calculated value could provide at least some useful level of regression testing. It would at least tell you that changing this value here is also going to change that value over there, which on a huge site could be ultra-helpful.
Selenium is about the best way to do this at the mo, I think. But Windmill (http://www.getwindmill.com/) could be there in time, I haven’t played with it properly yet.
Simon Scarfe - 14th October 2008
Interesting idea – this is something I’ve been thinking about recently with regards to performing automated regression testing on documents that represent all elements of a CSS system. I would be very interested to know how you get on with this and if you find a solution.
Natalie Downe - 14th October 2008
I’d love something like this!
It’s been on my mind for year or two now. Writing CSS is a truly one of the last refuges of Attempt Driven Development :-(
Another interesting thing might be a CSS/HTML-compiler, which, given CSS and some HTML, would generate IEHHbrowser-specific CSS-files and HTML-modifications automatically.
Antti Rasinen - 15th October 2008
@Simon,
Isn’t a screenshot-comparison tool inherently more of a regression-testing tool rather than a unit-testing tool?
Michael(tm) Smith - 15th October 2008
@Antti not sure about a cross browser CSS compiler. It would have to be a boat load of changing special cases which would be a nightmare.
@Michael true, but it would still be useful. Unit tests can be used for design but I think in terms of CSS the main use case is definitely going to be regression testing.
Gareth Rushgrove - 15th October 2008
Unless you develop a browser, CSS/rendering unit testing is a very weird rant. CSS is a text to describe your design, it has nothing to do with programming. Standard programming practices do not apply for CSS. CSS is a tool more for designers.
Frank, vizualbod.com - 28th October 2008
I dont see how CSS isnt an applicable target for unit testing. That its a declarative syntax doesnt change the fact that its code. But to test css effectively you cant have 2 moving targets. You need a static, known DOM which you test evolving css against, or vice-versa. Changing both and I think screenshot comparisons is the best you can hope for, and even then, presumably this stuff is changing to some purpose, so its possible the old screenshots are now out of date.
I’ve had some success manipulating either the CSS rule selectors or the DOM via javascript to be able to get before and after comparisons. I think this has to be a best-effort kind of exercise though, in which you attempt merely to reduce manual testing (read: head-banging) necessary by ruling out nasty stuff like syntax errors. You could perhaps flag overlapping selectors or duplicated rules. As much as I want this to exist, I’m struggling to come up with scenarios where it would be both practical and useful.
Sam Foster - 10th December 2008
Leave a comments