CRA Web Site Information Architecture Research - Usability Testing Phase 2

Prepared for the:
Public Affairs Branch
Canada Revenue Agency
FINAL REPORT
March 2008
POR# 303-07
Contract #46558-086882

Prepared by:
Phoenix Strategic Perspectives Inc

Le rapport complet en français sera fourni sur demande.

To request a full copy of this report, please contact Library and Archives Canada at:
613-996-5115 or 1-866-578-7777 or www.collectionscanada.ca

Media Enquiries:
Media Relations
Canada Revenue Agency
4th Floor 555 MacKenzie Avenue
Ottawa ON K1A 0L5
media.relations@cra-arc.gc.ca

Executive Summary

Background/Objectives

Phoenix SPI was commissioned to undertake qualitative research to assess the effectiveness of proposed changes to the information architecture of the CRA Web site. The primary purpose was to test a new navigation structure through usability testing exercises. The research was conducted using a mock-up of the CRA Web site that incorporated the new navigation structure.

Target Audiences

The target audiences were individuals (wage earners, seniors, and students) and businesses (small and medium-sized enterprises, self-employed individuals, and third party service providers).

Methodology

In total, 39 interviews (19 with individuals; 20 with business representatives) were conducted across three locations: Toronto, Montreal (French), and Vancouver. Fieldwork for the research took place January 28 to February 7, 2008.

This research was qualitative in nature, not quantitative. As such, the results provide an indication of participants’ views about the issues explored, but cannot be generalized to any of the populations included in this research.

Key Findings

Since similarities between the two main audiences involved in this research (i.e. individuals and businesses) outweighed differences, the executive summary will focus on both audiences at the same time instead of dealing with each audience separately. However, the body of the report provides detailed analysis of each audience on its own.

Completion of Tasks and Navigation Tendencies

As a group, individuals were more successful in completing tasks as part of the usability testing than business representatives. Most individuals successfully completed a majority of the exercises they attempted, while just under half the business representatives were able to successfully complete a majority of theirs.

Despite this difference, participants’ behaviour when completing exercises was very similar. For example, when looking for information, participants tended to focus on the middle of the homepage, with some explaining that this was sometimes why they did not notice certain links on either side of the page. Other common tendencies included assuming that the ‘Individuals’ and ‘Business’ labels are only headings, not functional links, and using the ‘Back’ button from the browser to move backwards in the site instead of using breadcrumbs or links in the left-hand menu. Both audiences included participants who typically tried to use the A-Z index on the main page in a first attempt to find information, and only tended to turn to browsing links when this was not successful. On the other hand both audiences also included many others who focused on browsing links and used the A-Z indexes only at lower levels of the site, or ignored them altogether. A key difference in behaviour was that business representatives, especially service providers, were more likely to say that under normal circumstances they would use the search engine on the homepage to find the information.

Organization of Information

To the extent that the organization or structure of the CRA Web site emerged as an issue during the completion of tasks, participants most often drew attention to the way the display or presentation of information affects or could affect the visibility of links (i.e. the likelihood of noticing them). Four organizational features of the site were often identified as affecting or potentially affecting the visibility of links: the tendency of the homepage design to draw one’s eyes to the middle of the page (making one less likely to notice other links on the page), the inclusion of links in sentences, pages with long lists of links, and information located ‘below the fold’ on Web pages.

Two other organizational issues were identified by participants as affecting access to information: the number of ‘clicks’ necessary to access certain information and the functioning of the A-Z index. Regarding the former, a number of participants felt that certain types of information could be located nearer the surface of the site (i.e. the homepage), thereby minimizing the number of clicks needed to access them. Regarding the A-Z indexes, there was a sense among some who used them that finding information through them might be the result of trial and error (i.e. trying different possibilities until the correct letter is chosen). Generally speaking, participants who used the A-Z indexes to locate information were successful in finding it. However, some were on the right track to finding certain pieces of information but had difficulty or were unsuccessful because they used a letter to search for information located under a different letter.

One organizational feature affecting the likelihood of finding information was identified only by business representatives. At least a few participants in each subgroup of business representatives (but primarily self-employed individuals) pointed to the issue of self-identification as something that affects the ability to find information. Specifically, some participants drew attention to the links under ‘Business’ that identify specific types of businesses (i.e. ‘sole proprietors and partnerships’, small and medium-size enterprises, and ‘corporations’). This caused a certain amount of confusion in trying to find certain types of information because some participants did not know how to self-identify – because it was unclear that a certain type of information was linked to a specific type of business, or because multiple categories applied to their business, so they were unsure which one to choose.

Clarity of Information

While lack of clarity was not identified as a widespread problem with the CRA Web site, most participants did encounter problems with clarity when trying to complete certain tasks. These problems manifested themselves in two key ways – vagueness and lack of precision. Lack of precision was the issue identified most often. Some labels and/or descriptions of categories were seen to be unclear because they did not adequately reflect their content, or suggest an intuitive path to certain information. While the labels or descriptions were clear in and of themselves, participants did not feel that all of them gave a clear or accurate idea of the content. Numerous examples are identified in the body of the report. On the other hand, some labels were described as unclear because they were considered vague, abstract, or indefinite.

Impressions of CRA Web Site

Overall impressions of the CRA Web site tended to be positive, and the focus of positive feedback was the site’s basic structure and organization. While there was widespread agreement that there is a lot of information on the homepage, this was seen by many as good because it makes information accessible at the surface level of the site, thereby minimizing the amount of clicking required to access it. Many also described the emphasis on topics for individuals and businesses as good, observing that this allows visitors to easily locate links they commonly use when visiting the site. Perceptions that the site is well organized were underscored by relatively widespread impressions that visitors will be able to find information they want or need.

There was general agreement that the links on the homepage are appropriately located, and most felt that the grouping of information under the various headings seems logical and complete. The one area where this was less likely to be the case was the left-hand menu of the homepage, which to some does not seem to be logically ordered, at least not in an immediately self-evident way. That said, most labels on the homepage were described as clear, easy to understand, and intuitive in the sense that they give a clear idea of the type of information to which they give access.

Suggestions

Participants collectively made numerous suggestions for making it easier to find information on the CRA Web site, although most were offered by no more than a few participants. Most suggestions were quite specific and focused primarily on improving the clarity of links and/or the descriptions accompanying them, making links more visible, and making information more easily accessible.

Conclusions and Implications

Despite differences in the exercises and their success in completing them, participants, regardless of audience, provided similar feedback regarding the CRA Web site and ways to improve it. Generally speaking, impressions of the site were positive in terms of ease of navigation, clarity of the labelling, and organization. To the extent that participants raised issues, the focus was on organization and clarity of labelling and descriptions. Participants offered numerous suggestions related to these issues for making it easier to find information on the CRA site. Moreover, these suggestions were concrete and precise, and most involve relatively small changes or adjustments. Particular attention should be paid to suggestions for improving descriptions of links since a key issue that arose as a result of the usability testing was lack of precision/accuracy in certain descriptions.

Date modified:

2008-07-30