Open Letter to ISO
Last week, Dr. Stuart Reid, convenor of Working Group 26 responded to the online petition calling for the withdrawal of ISO/IEC/IEEE 29119.
Dr. Reid focussed on a handful of the comments included with the petition, rather than the many, many blog posts that have been written on the subject in the last two months. As a consequence his response seems to have missed key aspects of the opposition to ISO 29119 and is potentially misleading. This article will restate and expand on the position of the ISST board.
On Opposition and Participation
In his response, Dr. Reid portrays a reasonable process whereby the testing community has been engaged in the development of ISO 29119, a process through which he claims objections have been fairly handled. He states that "development of the testing standards has been well-publicised worldwide"; that "workshops were run...where the content and structure of the set of standards were discussed"; that he invited "the broader testing community to comment on the standard".
These statements are true; what he doesn’t tell you about is the opposition he encountered and ignored. For example:
- At CAST 2006, James Bach and Dr. Cem Kaner debated Dr. Reid in opposition to commercial certification of testers using exactly the same arguments that we are now using against ISO 29119. James and Cem are two of the most well known testers in the world. They were the first in the industry to develop and describe exploratory testing. Their book Lessons Learned in Software Testing is a bestseller in the field. They founded the Context-Driven school of testing and the Association for Software Testing. Their vision of testing sharply contradicts the ISTQB/ISEB approaches. Yet despite several hours of debate, on stage and off, Dr. Reid told James that he did not understand the differences between Context-Driven testing and the ISTQB/ISEB, and that claims about such differences were simply marketing rhetoric. That dismissive and insulting claim was later repeated to a member of our board, Henrik Andersson.
- In 2008, during a presentation by Dr. Reid at the Teknologisk Institut in Stockholm, Henrik Andersson publicly challenged the notion of a software testing standard.
- At EuroSTAR 2008, Michael Bolton spoke about the dangers of standardisation in his presentation "Two Futures of Software Testing". Dr. Reid was in attendance.
- At Agile Testing Days 2010, both Michael Bolton and Elizabeth Hendrickson argued against certification and standardisation with Dr. Reid.
Opposition was and is well known to other members of the Working Group. In the abstract for his CAST 2012 presentation, Jon Hagar (one the editors) admits as much: "I find myself in a position of doing something that some will see at odds with the context driven test community”. Hagar further states, in an email to James Bach, "the standard 29119 was going to be produced no matter what anyone did". It would appear that despite known opposition, the convenor and his Working Group intended to pursue their standardisation agenda regardless. They continue to refuse to take opposition seriously: in a recent comment on Michael Bolton’s blog, Anne Mette Hass (another member of the Working Group), she said she had “better things to do” than reply to challenges to the ‘standard’.
Reid seems to fundamentally misunderstand the nature of the opposition to 29119. His answer was to invite people to participate, and to declare that objections can only be handled "via the ISO/IEC or IEEE processes". There is a simple reason why many of us did not participate in the development of this ‘standard’: we, and many of our signatories, do not believe that it is appropriate to standardize testing. This is not a matter of content, but of principle: there is a struggle going on in our craft amongst different and competing visions of testing. This has been playing out mostly in the free market and in our contrasting conferences. No party to this debate can claim to own the testing craft. Therefore we do not believe in having a software testing standard at this time. This is not an indication of assent or apathy in our community: if you believe that violence against children is wrong, you are unlikely to participate in the development of a standard for the administration of corporal punishment in schools.
Reid also argues that the horse has already bolted: "a petition initiated a year after the publication of the first three standards...represents input to the standards after the fact and inputs can now only be included in future maintenance versions". As he misjudged the opposition, he may also have misread the petition; it calls for withdrawal of parts 1 to 3 (those that have already been issued) and suspension of the remaining parts. It does not request modification, amendment, or the inclusion of any change: it calls for withdrawal. There is nothing to stop ISO from doing so, other than their desire to claim it as a standard.
He further argues that, were the ‘standard’ available at no cost and thus more accessible, "many of these people would have refrained from ‘signing’ the petition". That is speculation and, unlike ISO 29119, speculation is free. For example, we might speculate that if 29119 were freely available, if more people had the opportunity to review its content, then more people would have signed the petition.
As those who participate in developing a standard are likely to be predisposed to standardisation, systemic bias in the standards process is a strong possibility. Do ISO have any evidence that their process accounted for safeguards against systematic bias and self-selection of the Working Group?
There is also the strong appearance of a conflict of interest, in violation of the ISO Code of Ethics, a code which is intended to prevent "conflicts of interest by communicating in a fair and transparent manner to interested parties when work on new standards is initiated and subsequently on the progress of their development, ensuring that market needs are the driver for the development of standards". Thus there is an appearance of a conflict of interest, due to personal antipathy as well as commercial interest, on the part of the Working Group for ISO 29119. It has led to the systematic disregard for the work of prominent testing thinkers who strongly disagree with the ‘standard’.
We would like to believe that an organization such as ISO has rules and procedures for this, such as a register of members’ interests, a procedure for declaring conflicts of interest and a record of any instances in which a conflict of interest has been declared. We call on ISO to release these procedures and records into the public domain so that potential customers can form their own opinions. We call on ISO to publicly investigate whether its code of ethics and code of conduct have been violated.
In order to establish credibility, Reid makes an appeal to authority: he points to the expertise, the "number of years’ and range of testing experience", of the Working Group. Yet it is unclear how these experts are selected. Are there clear selection criteria, and if so what are they? Are they selected because they are like-minded individuals who support standardisation in testing? Or are they self-selected, volunteering because they want to see testing standardised?
Participation in an ISO committee confers no special insight. Who is to say that the experiences of his group are any more valid than that of the thousand-plus signatories of the petition? Who is to say that their experience is greater than the C-levels, directors, testers, test managers, consultants, academics and former auditors who have voiced their opposition? Who is to say that their position has the greater validity? And what evidence is there that the Working Group’s expertise has resulted in an effective set of practices?
Reid makes a second appeal to authority when he points to an academic study (Kasurinen, 2011, Software Test Process Development) concerning the process model outlined in ISO 29119 Part 2.
Reid claims that this study shows that:
- "The concepts presented in the ISO/IEC 29119 test process model enable better end-product quality"
- "The ISO/IEC 29119 test standard is a feasible process model for a practical organization with some limitations"
- "The ISO/IEC 29119 test standard is a feasible foundation for a test process development framework"
The first claim misrepresents Kasurinen’s position. Kasurinen surveyed a number of organisations and concluded that those with processes similar to 29119 had greater confidence in their software quality, not necessarily better quality. The organizations surveyed had not implemented the ‘standard’, and confidence does not equal quality. How did Reid draw such an inference from Kasurinen’s research?
With regards the second and third claims, there may be threats to the validity of these conclusions. Again, the organizations surveyed had not implemented 29119. Some of those organisations also reported 29119 to be document heavy and overly focused on organizational policy and strategy, and providing little guidance on implementation. Further, does “a feasible process model” or “feasible foundation for test process development” necessarily equate to a feasible foundation for testing? Given these points, it seems a stretch to claim feasibility. Even if we accept Kasurinen’s conclusion, feasible does no equate to worthwhile. Just because you can do something, doesn’t mean that you should.
Dr. Reid’s use of Kasurinen’s research seems both inaccurate and uncritical. Did the Working Group review the study or subject it to any kind of scrutiny? Perhaps this is why, in his 2014 presentation at ExpoQA in Madrid, Reid cited "no evidence of efficacy" as one of the factors most likely to stop ISO 29119. Has it undergone field testing at all?
So, what further research do the Working Group have to offer? It is interesting to note that the above research was conducted after 29119 was drafted. What research, if any, is the ‘standard’ actually based on? It makes many bold claims, such as testing without first having established test policies and strategies for the organization as a whole "gives less coherence to the testing...and typically makes the testing performed in projects less effective and efficient". It outlines many such ‘good practices’ without providing a single citation. We would welcome a debate based on evidence. ISO: publish your citations; show us your evidence for each proposed practice. Publish your review records, and show how each claim was scrutinised, how each point was rigorously debated. In addition, were 29119 a new drug there would have been testing to establish what side effects it might cause. Given the potential for harm, what testing has been performed to assess the possible negative consequences of adopting it? This should be disclosed.
Reid makes much of the ‘standard’ not being compulsory. He tells us that ISO's definition of a standard is: "guideline documentation that reflects agreements on products, practices, or operations by nationally or internationally recognized industrial, professional, trade associations or governmental bodies".
Unfortunately that isn’t ISO’s only definition. According to their website: "A standard is a document that provides requirements, specifications, guidelines or characteristics that can be used consistently to ensure that materials, products, processes and services are fit for their purpose". So yes, a standard might be a guideline, but it may also be a requirement or specification as to how you are expected to work.
Regardless of how Reid imagines 29119 will be used, we worry about how others might choose to use it. When a company, their customer or regulator seizes on 29119, sees that it is an "internationally agreed standard" and takes that as a convenient (if misguided) proxy for testing quality, we would expect compulsion to follow. Reid acknowledges this when he says "if specified in a contract then they [the standards] can define requirements on the testing".
Reid tells us that there is "no link between the ISO/IEC/IEEE Testing Standards and the ISTQB tester certification scheme". This does not appear to be the case:
- A number of members of the Working Group, including Dr. Reid, have affiliations with the ISTQB, its member boards, or related service providers.
- 29119 makes extensive use of the ISTQB glossary and, for anyone who has been exposed to the ISTQB syllabi, makes for a familiar read.
- Anne Mette Hass has stated “ISO 29119 is based on the ISTQB syllabi, and, as far as I understand, it is the intention that the ISO 29119 testing process and testing documentation definitions will be adopted by ISTQB over time.”
- A source close to the Working Group has advised us that liaison between the ISTQB and Working Group 26, with the goal of aligning the ISTQB syllabi to ISO 29119, has been discussed on several occasions.
We can easily imagine a near future where the ISTQB have aligned their syllabus to 29119, and are the de facto choice for standards-compliant tester training. Whilst the ISTQB is a non-profit organization it supports an ecosystem of training and examination providers, many of whom are represented on its member boards. If there are links between the organizations, what implications does this have for the independence and impartiality of the Working Group?
Certification of individuals is one thing: certification of companies, i.e. registration ISO 9000 style, is perhaps a greater worry. Like ISO 9000, 29119 has been designed so that organisations can claim compliance. Several of us have first hand experience of organisations that have sought compliance with some form of standard, external or otherwise: the effort required to demonstrate compliance, or justify waivers in the case of tailored compliance, can be significant - an opportunity cost that threatens the quality of testing.
Much like the ISTQB, ISO 9000 supports an ecosystem of commercial interests: certification bodies, consultants, training organizations etc. We fully expect to something similar to evolving around ISO 29119, indeed, our sources tell us that the Working Group have considered developing relationships with such third parties as a strategy for marketing the ‘standard’.
Reid employs some interesting arguments to defend against the claim that 29119 is incompatible with Agile. "The standards were being continually updated until 2013 and so are inclusive of most development life cycles", in other words it isn’t old fashioned by virtue of it having been written recently. Being written recently does not imply that the authors understand Agile or took care to incorporate Agile test approaches in their ‘standard’. "The test documentation standard...is largely made up of example test documentation and for each defined document type example documentation for both traditional and Agile projects is provided", in other words you can use it with Agile because we’ve given examples of how to fill out documents. This seems incongruous to us.
As Reid indicates, you can tailor out aspects of the ‘standard’ that don’t apply to you. In an organization seeking compliance, all you need do is justify, document, and gain approval for each exception, for each clause or "shall" statement that is not being adhered to. ISO/IEC/IEEE 29119 Part 2 contains 77 of the former and 123 of the latter. This does not seem Agile to us.
Reid responds to the suggestion that the ‘standard’ does not support Exploratory Testing with the claim that it "is explicitly included as a valid approach to testing in the standards". Unfortunately the authors of 29119 do their best to invalidate ET. Their comparison of scripted and unscripted testing (part 1, clause 5.6.6) displays the kind of misconceptions about ET that are common amongst those who have not given it anything more than casual consideration. For example, 29119 claims repeatability, reuse and traceability as advantages of scripted testing (and implicitly as disadvantages of unscripted testing). We won’t argue the merits of repeatability and reuse here - that is a lengthy conversation; suffice it to say that by citing these so-called weakness the authors display ignorance of how the practice of exploratory testing has developed over the last decade. They make the mistake of imagining scripted vs. unscripted testing to be a dichotomy rather than a continuum, and in doing so fail to realise that ET need be neither unplanned nor undocumented.
Reid further goes on to stress the implied inferiority of ET by quoting directly from 29119: "When deciding whether to use scripted testing, unscripted testing or a hybrid of both, the primary consideration is the risk profile of the test item. For example, a hybrid practice might use scripted testing to test high risk test items and unscripted testing to test low risk test items on the same project". We believe Reid has this backwards: if we were testing something high risk, we'd much rather have testers applying skill and judgement to Exploratory Testing than blindly following a set of pre-ordained instructions.
When it comes to context, Reid’s defence is no better. He claims that he "fully agree[s] with the seven basic principles of the Context-Driven School", yet his arguments suggest otherwise. He notes that "we do not claim that these standards define ‘best practice’" but rather provide a "definition of good practices in the testing industry". This is a dodge. Calling something a ‘good’ practice is no better than calling it a ‘best’ practice; no practice is universally good across all contexts.
Reid reverts to best practice assertions in his discussion of risk based testing: "the standards do, however, also mandate that a risk-based approach is used". This amounts to a claim that risk based testing is best in all circumstances. We disagree. One of our board, Iain McCowatt, has tested in contexts where risks were irrelevant, for example in testing a legacy system to determine its behaviour, as a baseline for reverse engineering.
ISO/IEC/IEEE 29119 has been created despite known opposition; there appears to be little, if any, evidence to support its efficacy; it appears to be the product of a process that was subject to systemic bias and commercial interests; it misrepresents some of the most significant recent developments in the craft. We reject the validity of ISO/IEC/IEEE 29119 as a standard.
We call on ISO to require Working Group 26 to publish citations to any research upon which 29119 was based – including any research that was rejected; to publish any evidence of a need for a testing standard; to publish evidence of any field testing that was conducted and to publish their review records; to publish all records related to member interests and instances of declared conflicts of interest; to investigate whether their code of conduct or code of ethics has been violated. We call on ISO to withdraw these ‘standards’ until such a time as a case for standardization can be made that both withstands scrutiny and can obtain consensus within the testing profession.
- Ilari Henrik Aegerter, Henrik Andersson, Johan Jonasson, Iain McCowatt