Measuring the Impact of Communications on Atlantic Canadian Homeowners' Attitudes towards Underground Economic Activity - Phase II Post-Campaign Test

Prepared for the:
Canada Revenue Agency
July 2011
Contract # 46558-111445/001 CY
POR 058-10
Contract Award Date: 2010-10-21


Prepared by:
Corporate Research Associates

Media Enquiries:
Media Relations
Canada Revenue Agency
4th Floor 555 MacKenzie Avenue
Ottawa ON K1A 0L5
media.relations@cra-arc.gc.ca

Executive Summary

Context and Methodology.

The Canada Revenue Agency (CRA) sought to understand the impact of the Agency's actions on future compliance behaviour in the underground economy in Atlantic Canada. The Agency developed a broad communications strategy, referred to throughout this report as an "advertising campaign," that targeted homeowners in Atlantic Canada as part of the Agency's overall underground economy strategy. As part of measuring the impact of the CRA's actions, a two-phased research endeavour was undertaken. Phase I provided baseline assessments on several indicators prior to the implementation of the advertising campaign. Phase II involved reassessment on the same indicators to determine change attributable to the advertising campaign.

In this context, the CRA commissioned Corporate Research Associates Inc. to undertake, at a total project cost of $112,897.48 (includes HST), the Measuring the Impact of Communications on Atlantic Canadian Homeowners' Attitudes towards Underground Economic Activity – Phase II – Post-Campaign Test study.

A primary objective of the research was to measure the impact of the advertising campaign on the attitudes of Atlantic Canadian homeowners towards tax non-compliance in the construction and home renovation sectors. This report examines the results from the Phase II post-campaign test of the research, primarily entailing a comparison between Phase II and Phase I (pre-campaign) findings. Phase II consisted of two waves of telephone interviewing: one (Wave 3) was conducted immediately following the advertising campaign to assess immediate campaign impact, while the other (Wave 4) was conducted approximately two months after the completion of Wave 3, so as to assess the sustained or potentially delayed impact of the advertising campaign. In total, 3,150 interviews were conducted in Phase I, and 3,155 interviews were conducted in Phase II. The data collection periods were as follows:

• Pre-Campaign: Wave 1 was conducted between February 25 and March 24, 2010, and Wave 2 was conducted between April 6 and 28, 2010.
• Post-Campaign: Wave 3 was conducted between November 3 and 26, 2010, and Wave 4 was conducted between January 28, 2011 and February 15, 2011.

A cross-section of randomly-selected homeowners in major Atlantic Canadian municipalities was surveyed at each data collection time period (please refer to Appendix A for a copy of the survey questionnaires). These Atlantic communities served as the study's "treatment" group, as it was in these areas that the advertising campaign was administered. In addition to surveying in Atlantic Canada, five communities outside of Atlantic Canada were selected to "match" the Atlantic Canada municipalities (excepting one Atlantic Canadian community, for which no match was found), and these matched communities were surveyed during the same time periods (please refer to Appendix B for an explanation of the "matching" process). These communities served as the study's "control" group when examining the impact of the advertising campaign, as no CRA commissioned UE-themed advertising campaign was administered in the locations. In comparing Phase I results with Wave 3 and Wave 4 results, analyses focused on evaluating the advertising campaign's impact (please refer to Appendix C for a copy of the study's tabular results). This involved examining changes between Phase I and Wave 3, and between Phase I and Wave 4, to determine if such changes were significant and in the expected direction. Evidence of advertising campaign impact would be, for example, significant changes (in the intended direction) in the pattern of responses for Phase I Atlantic homeowners as compared to Wave 3 Atlantic homeowners, while the pattern of responses for Non-Atlantic homeowners remains consistent between Phase I and Wave 3. Given that the data for this project were collected via random sampling, the results of this study can be generalized to the overall populations of homeowners within the surveyed communities, recognizing the limitations presented by the study's margins of error and confidence intervals.

Results.

A primary research objective was to ascertain whether the advertising campaign influenced attitudes of Atlantic Canadian homeowners in the treatment communities. An examination of the thirteen survey indicators deemed by the Canada Revenue Agency as critical to evaluating the impact of the advertising campaign identifies few differences post-campaign compared with pre-campaign.

In this assessment, thirteen of the survey questions regarding homeowners' attitudes towards tax non-compliance in the construction and home renovation sectors were deemed by the Canada Revenue Agency as critical indicators of the impact of the campaign. These thirteen measures are identified throughout the detailed analysis via bolded italics script, and are also catalogued in Appendix D.

Among these thirteen critical indicators, results show a limited number of statistically significant/marginally significant differences [Footnote 1] post-campaign compared with pre-campaign, in the direction intended by the Canada Revenue Agency:

• If homeowners do not have a signed contract, they put themselves at risk if a worker is hurt (statistically significant increase between Phase I and Wave 4);
• The understanding of the lack of protection for homeowners in the event of poor quality work if there is no signed contract (marginally significant increase between Phase I and Wave 3, and between Phase I and Wave 4);
• Disagreement that if a homeowner avoids paying taxes, no one loses (marginally significant increase between Phase I and Wave 3); and
• The likelihood of hiring someone under the table for a $10,000 job in the future (marginally significant decrease between Phase I and Wave 3).

Analysis of the average number of preferred responses provides limited support for the advertising campaign having the intended impact. Specifically, the advertising campaign was found to have had a marginally significant impact on Atlantic Canadian homeowners' average number of preferred responses to critical questions between Phase I and the immediate post-campaign assessment (i.e., Wave 3). The advertising campaign was not found to have had a significant impact, marginal or otherwise, on Atlantic homeowners' average number of preferred responses to critical questions between Phase I and the delayed post-campaign assessment (i.e., Wave 4). The findings suggest that further research is warranted into the potential effects of advertising on Canadians' attitudes towards tax non-compliance.

The study results also indicate selected statistically significant/marginally significant differences in non-critical indicators:

• Awareness of the underground economy (statistically significant increase between Phase I and both waves of Phase II); and
• The CRA is doing a good job of advising the public about tax obligations and requirements (statistically significant increase between Phase I and Wave 3, and a marginally significant increase between Phase I and Wave 4).

The research indicates that the advertising campaign reached a large segment of Atlantic homeowners in the target communities. Specifically, seven in ten Atlantic homeowners recall seeing, hearing, or reading advertising about the risks associated with paying "under the table" for home construction, renovations, and repairs immediately post-campaign (Wave 3). This proportion drops to approximately five in ten during the delayed assessment (Wave 4). The advertising campaign received positive evaluations, but this finding should be interpreted in light of the finding that Non-Atlantic homeowners who indicated they recall such an advertising campaign, also provided positive evaluations. Finally, from Phase I to Wave 3, as well as from Phase I to Wave 4, there was an increase in the percentage of Atlantic Canadian homeowners who believe it is common for Canadian homeowners to conduct home construction by way of the underground economy.

Footnotes

[Footnote 1]

Statistical significance is defined as a result for which the obtained p-value is equal to or less than 0.05, while marginal significance is defined as a result for which the obtained p-value is less than 0.10, but greater than 0.05.

Date modified:

2011-08-16