22 January 2015

The exponential cost of testing

As any software project continues it gains features. Very few projects shrink over time, its usually an ever increasing list of features and an ever growing codebase. Most software projects last years and many companies that invest in software are doing so for the long term. There are some exceptions, sometimes companies are creating it to throw it away but most of the time companies are only intending to build a system once and to do so with the intention of running that service for a long time. There is an unfortunate growth in testing effort that comes along with features that you simply can't escape however.

We have to assume that our overall goal of testing a product before release is to determine it has no defects. We are always going to miss cases but the purpose is to ensure the quality of the software product is as good as it can be. When we add functionality the total amount of testing that the product now requires increases. We can make arguments for whether adding that functionality causes a linear growth in the total testing effort, whether its exponential because features interact with each other or sub linear because the features have an overlap. But I think its fair to say we all expect adding a feature to add additional testing cases, which results in the following 3 possibilities for the growth of total testing effort to validate a release in relation to features/time.




Given this progression of effort over time the solution proposed in the late 1990s was to automate most of the test cases. Humans aren't very good at running the same test over and over, they tend to skip scenarios and make mistakes and its incredibly boring and labour intensive to execute testing scripts against a piece of software. More importantly since the testing effort grows as the project continues a project only really has two choices. One is it keeps adding staff such that the testing team is big enough to test everything or it chooses to avoid testing some of the system. Every company I have seen has chosen the skip tests option, and indeed I don't think we see any big companies today that are now 99% testers having pursued the test everything approach. So the strategy of only testing some items usually falls down to testing only the latest features and maybe some existing ones that interact with those new feature. While a reasonable strategy for adhoc testing it is not a very rigorous way to ensure software quality and bugs in older features impacted by the new features will be missed. Indeed if a project team is refactoring and there are no automated tests to run its extremely likely that bugs will be introduced and not detected with this testing strategy, which is why so few code bases without automated tests refactor their code. It is far too expensive to refactor when a complete validation of the system will be required.

So then Test Driven Development came a long and we had the discipline within teams to ensure that this progression of testing effort was passed onto the machine. It took time to develop the tests but the end result was that the testing team didn't need to gain in size every release or risk missing parts of the system. It was without a doubt much cheaper to maintain the system, it was possible to refactor safely with confidence that the system wasn't now broken and hence the code didn't rot as quickly. The system could be released more frequently and with good confidence it worked and all the while the team size could remain constant and economic.

Yet today in multiple blogs this practice is vilified as slowing down developers and projects. Maybe these are the short projects that don't need to think about testing effort for the long term and don't need to worry about refactoring as they are throw away. Or perhaps they really do have a product that will be made and shipped just the once such as an embedded system. But it seems to me that many developers are talking themselves out of the basic problem, that they don't want to write tests and hence they are useless. But while they have all these other drawbacks and benefits is is not just about the developer, its about the teams effort as a whole and hitting a certain level of quality. This continuous growth in effort is a real problem. If you want to release a new version every month you either take the risk of releasing a bad release and leave the bug testing to your customers, or you have an enormous testing team or you have automated tests. If you take that down to weekly or even daily releases it becomes impossible to do it manually on all but the smallest of software projects and enormous testing teams. This relationship isn't going away any time soon, you always have to make that choice of risk of releasing a bug again for something that would have easily been caught had the feature been tested properly or automating the test to begin with. Its why I remain firmly in the belief that automated testing is critical for a projects future productivity and quality. Without it velocity has to slow in the future as bugs are fixed and release quality steadily drops either with old code rotting or new code breaking. TDD is about discipline to do this so that it always happens. Because its no surprise that developers don't like to write the tests once the code is done, its just like documentation in that regard and if its left to later then later becomes never.

19 August 2014

The mentality of programmers

The people who enjoy programming are those that can sit there for a day while the computer constantly tells them they are wrong, then right at the end of the day as they finally make their program work they internally say "I am awesome" (despite all the evidence to the contrary).

14 July 2014

4k verses 1440p and gsync the next big trade off

For the last 3 or so years there has been a classical trade off in the monitor world. Monitors have a variety of different technologies that determine the characteristics and you have had to choose between them. You either chose to get an IPS based screen and got high colour accuracy but at the cost of more ghosting and stuck at 60hz or you could get a TN based screen which had lower ghosting, could do 144hz but lower resolutions and worse colour accuracy. Thus the monitor world was split into professional grade screens and gaming screens. Which as a programmer and gamer meant I always had to choose one or the other.

CES in January showed us that pretty much everyone is now talking about 4k, gsync and freesync panels. 4k (UHD) is all based on an enhanced version of TN and so are the gsync panels. I don't quite know what happened to colour accuracy in all this but the manufacturers either managed to make TN produce a good colour image or they just stopped bothering worrying about it at all. Suffice to say a lot of graphics professionals are not going to want any of the 4k panels on offer when they don't produce print quality matching colour. Windows applications still aren't ready for the scaling necessary for 4k panels, even chrome doesn't yet cope with it and so I don't think Windows 8.1 ecosystem will be the one that 4k succeeds in, the software isn't ready. You could buy one and find out for yourself how annoying icons 1/4 of the size in some applications but not others is, but I think its best to wait until Microsoft and the software companies have got everything working better.

Gsync is a great advance for gamers and movie watchers because its going to fix the classic cadence problems both these applications have. Its looking like its going to release in the next two weeks properly rather than the module release that happened in January. Most of the gsync monitors are going to be 1080p and carry a big premium and likely aren't any more advanced than a current TN 144hz monitor with gsync added. Gsync is likely worth it on its own for a gamer but they really don't fix the classical trade off between professional screens with higher resolutions and refresh rates and blur, its firmly in the gaming camp. Asus however has the ROG Swift PG278Q coming out on the 28th July (if the scan.co.uk preorder page is to be believed). Its a 27", 1440p gsync based monitor, with 144hz and 8 bit colour. No TN screens have 8 bit colour, its always been 6 bit + FRC at high refresh rates and this monitor seems to do it all. If it can get 99% SRGB colour coverage (rather than the 80% or less most TN panels get) its going to look absolutely fantastic for a gaming monitor. The high refresh rate is really nice and then on top of that its 1440p which is about the sweet spot for higher density until Windows applications scale better. It could be the perfect trade off monitor.

Later this year there will be a 4k gsync monitor. The problem is that none of the cable standards today actually support 120+ hz on 4k resolutions at 8 bit colour. This is a real problem because I am sure a manufacturer out there has realised that a lot of gamers would want to play at 4k for the increased fidelity and do so at higher refresh rates if possible, but alas the standards for that amount of bandwidth just aren't ready yet. So the next generation of trade off is looking a little different from before:

1) The old IPS screens with 60hz. 8-10 bit colour and low density (<100 dpd="" p="">2) The new TN 4k monitors with 60hz, 6 bit colour but high density
3) The gsync monitors 1080p with 144hz 6 bit colour, low density
4) The gsync Asus ROG Swift 1440p with 144hz, 8 bit colour, medium density

That Asus monitor just stands in a class all of its own for not only being gsync based, higher resolution and 144hz but also that 8 bit colour panel just tops it off. I can't wait to see some reviews and I really hope they got the colour accuracy, anti ghosting technology and latency right on this one. It might be the first time since LCD displays were released that we have a genuinely low latency decent gaming monitor that also produces good colour quality. I can dream.