Improving the quality of parliamentary reporting: from peer review to ensuring customer satisfaction

Citizen as Customer

If you buy a Creuset cast iron pan, you buy high quality. Such a pan has been checked no less than 30 times during the production process, the manufacturer claims. After every mechanical action, there is an employee who checks whether the product is in order. Overkill? Apparently not when you are in the business of making a profit.

Anyone who uses a government service also expects high quality. In the (digital) business world of recent times, the appreciation and experience of the customer has been central. However, in civil service circles, there has been some resistance to treating citizens as customers. The experience that the Dutch Parliament recently had with a customer panel of the deaf and hard of hearing about live subtitles may change their minds.

Quality Control – The Traditional Approach

A citizen who reads Hansard also wants high quality. In the normal routine of the Parliamentary Reporting Office of the House of Representatives of the Netherlands—the PRO—a report is checked three times during the production process: by the reporter, by the proofreader or corrector, and by the speaker. Before, that always seemed sufficient.

Nevertheless, professional reporting can always be improved, as was shown by the Telegraaf incident of 2013. In a high-profile article, this newspaper claimed that the President of the Parliament had censored Hansard. That was by no means the case, but a passage had been omitted by a reporter who had used the editing knife too boldly. This honest mistake showed the unintended and unexpected side effects that can occur. This is increasingly the case in today’s media age, in which Hansard can be compared with video or sound recordings with great ease by anyone who has access to the internet.

Checking Reports Against The Audio

To prevent errors, such as in the Telegraaf incident, the PRO considered supplementing the existing routines on quality and the usual quality assessment by the proofreader or corrector and the speaker. In the ensuing discussions about editorial freedom in work meetings, it was suggested not only to read but also to listen. Extra control was exerted on quality.

However, it was deemed not feasible for the reader or corrector to take on this task; the structural monitoring of turns requires extra manpower. A working group was installed, which introduced a new routine in the same year. It introduced a method of peer evaluation to the office in which all reporters annually check each other’s work against the audio recording. In this process, the summer recesses are used to have a large amount of reports by all reporters checked by their colleagues. This routine was explained at the IPRS meeting at the 51st Intersteno Congress on 23 July 2017 as “a truly horizontal and bottom-up system of monitoring quality. And because of the opportunity to comment on the turns anonymously, this system offers comfort to the reporters and creates a safe space for constructive criticism and feedback … The goal is to have the reporters draw their own conclusions to improve their work.”(Goorden, Mulders & Parren, 2017)

Initially, there was great reluctance to scrutinise the work of colleagues seriously. However, it later became a valued practice. Reporters learn from each other, from each other’s mistakes and from each other’s suggestions. In terms of quality control, or rather quality improvement or awareness of quality, there is certainly added value.

Peer review is thus a big step forward in ensuring high quality in parliamentary reporting. None the less, it remains a traditional solution because it involves professionals who, in order to do their job better, look at the work of other professionals. More specifically, peer review does not allow us to see the view of the customer. Since the customer has been central to the business world for many years, it is imperative that the “citizen as customer” concept enters the world of the Civil Service as well. The Government are not a business. They are not in the business of making a profit. However, putting the customer first can perhaps even be the way substantially to improve the quality of services or products in the public sector.

Customer First – Take One

Some 20 years ago, the PRO attempted to gain the reader’s point of view by conducting a customer satisfaction survey. The survey was submitted to MPs as the first users of Hansard and those with the formal right to suggest corrections. It is known from this survey that less than half of the 150 MPs took the trouble to respond. The result of the survey was that, overall, MPs were satisfied with the reporting. The poor response rate was probably at least partly caused by the fairly general question.

At the time, it was exciting to ask MPs about their satisfaction with the official reports, as the consequence or effect of a possible negative assesment was unknown. However, one could ask why this question had to be asked, since making Hansard was and is a legal obligation for the PRO. Also, fear of a critical response led to asking a question that was too general, with an answer that gave no direction. That conclusion was underscored by the fact that MPs make up only a small fraction of the readers of the reports.

Customer First – Take Two

In early 2020, the PRO started with a more deliberate attempt to involve the customer in a relatively new service of the PRO: live subtitling. This service is aimed at the deaf and hard of hearing. The subtitles are also used by employees who work in a quiet office environment. In a natural way, the customer was thus not limited to MPs.The service is provided by reporters of the PRO who have received professional training. As of 2016, websites and applications in the European Union have to comply with the web content accessibility guidelines. In 2016, a European directive entered into force, which was translated in the Netherlands into the temporary decree on digital accessibility for governments. The decree was made on 3 May 2018 and contained temporary rules regarding the accessibility of the websites and mobile applications of government agencies. This decision entered into force on 1 July 2018 and also applies to the House. The websites of the House of Representatives and other applications have been independently inspected for accessibility in accordance with the WCAG21-AA standard and passed.

However, the inspection described above is the formal review. Digital accessibility is about more, namely whether the standard mentioned is also sufficient to give everyone access to digital information and services. This can be better attained by supplementing the formal assessment with a qualitative assessment by a customer panel of people with a disability. After some discussions with representatives of these customer groups, the PRO has chosen, for practical purposes, to organise the customer panel on live subtitling not nationally but locally. Moreover, it was decided not to do it ourselves, but to use a local foundation for residents with a disability and with experience of customer panels. The aim of this pilot is to optimise the use of and access to the live subtitling of the House of Representatives by listening to the potential users of the service.

The pilot is still ongoing. It has turned out to be a huge challenge to organise the target group—the deaf and hard of hearing—for live subtitling in a customer panel, even if you make use of the necessary experience. For example, the use of sign and writing interpreters is costly but crucial. A new challenge came on top of that in May 2020 with the Covid-19 outbreak. Strict measures have been taken in the Netherlands, as in other countries, which means that meetings can be held only with due observance of social distancing. A number of participants also indicated that they avoided meetings altogether because of their poor health and all the associated risks. In practice, this means that all testing and communication with the customer panel are proceeding via video meetings—an extra effort for all participants, including the sign and writing interpreters.

Conclusions

The ongoing pilot with a customer panel has revealed a considerable gap in perspective between the views of the professional subtitler and those of the deaf or hard of hearing. Here the professional and civil servant, with his or her pride in providing the service, faces the deaf or hard of hearing, for whom this is nothing less than the realisation of their right to participate in society by gaining live access to parliamentary debate—a clash of perspectives indeed.

A cautious conclusion, for the time being, is that where a service contains a fundamental right, the quality expectation is much higher than in the case of offering a service or product on the market. Anyone who organises such meetings for the first time would do well to prepare the professionals and civil servants involved: the Dutch experience shows that the message might be conveyed by the target group in a very direct manner. That said, this collision of the perspectives of the professional and the customer will help to develop the service rendered. Even though the process is still unfinished and we are waiting for the report from the customer panel with recommendations on how to improve the live subtitles, we can confidently say that working with customer panels will be essential for promoting quality in the future.

Henk-Jan Eras is a Quality Officer with the Parliamentary Reporting Office of the

House of Representatives of the Netherlands and a member of the TIRO Editorial Team.

References

Goorden, F.; Mulders, I.; Parren, S. (2017). Checking turns against the audio–a system of peer evaluation in the Dutch Parliamentary Reporting Office. 51st Intersteno Congress, 23 July 2017.

Web Content Accessibility Guidelines, latest version (WCAG21-AA): https://www.w3.org/TR/WCAG21/