Satisfaction Survey by Web or by Paper? A Case Study at a Fortune 500 Company

By Brian Laughlin¹

Surveys are an invaluable tool for helping organizations stay abreast of issues within the corporate social environment. As the cost and pace of daily business increases, so does the need for high quality, cost-effective, timely, and efficient data collection. Traditionally, surveys have been done using paper methodologies, which can be costly to administer in terms of time, labor, and materials. The advent of the Internet has dramatically increased the ease, reach, and speed of survey administration and data collection, while simultaneously decreasing associated costs, making surveys faster, easier to administer, and cheaper than ever before. While paper and web methods for disseminating surveys have been thought to produce similar results, there is some evidence suggesting that people may respond differently, depending on the modality of administration. Consequently, the equivalence between the two should not be taken for granted.

The purpose of this study was (i) to evaluate the psychometric structure of an employee opinion survey tool, and (ii) to investigate the degree of equivalence of the same tool administered via web versus paper and pencil. The field data was collected from 8,509 employees, which were examined by both methods, at large Fortune 500 industrial corporation.

Materials

The data for the field study was collected using an Employee Opinion Survey, with the purpose of providing the company with employee feedback on management, work environment/atmosphere, and the company in general. This comprehensive survey was comprised of 44 Likert-scale questions broken into 9 distinct areas: Velocity and Urgency, Continuous Learning, Working Conditions/Health and Safety, Team Culture, Product and Business Performance, Management and Leadership, Integrity/Diversity, and Overall, and concluded with a free-response area for comments.

Each survey was serialized by either a serial number on the paper and pencil surveys, or a unique pin number for the web surveys, in an attempt to avoid multiple responses from any given user. In order to assure employee anonymity and survey confidentiality, surveys were collected and analyzed by an outside contractor who specializes in corporate surveys.

Results

Psychometric analyses of the survey responses demonstrated the survey to be both sound and internally reliable. An exploratory factor analysis confirmed the existence of nine factors. Results from a discriminant analysis that compared the factor structure of the web and paper surveys indicated no discrimination between the two modes. There were, however, substantial differences found in responses from employees paid Hourly or by Salary.

The most discriminating of the survey's sub-scales between the Hourly and Salaried Employees were Continuous Learning, Team culture, and Integrity/Diversity. The responses were most divergent in these areas as well. Below are summaries of the most discrepant survey constructs, main issues of dissatisfaction, and explanations for the responses of Hourly versus Salaried employees.

Hourly Employees

Continuous Learning: Opportunities and assistance with education.

Many hourly jobs are narrowly focused and repetitious in nature. Also, due to production deadlines, there may be little opportunity for employees to work cross functionally, innovate new processes, or cross train to learn new jobs. This may be perceived as being held back by the employee.

Team Culture: Empowerment and recognition in relation to accomplishing job tasks.

Hourly employees appear to feel that their work is not valued, and that they are not enabled to work cross functionally. This again may be due to the often highly specialized jobs characteristic of hourly paycodes. In addition, production metrics are focused on the ability of the shop to produce, while their union contract strives for fairness, by treating everyone equal, with advancement based solely on years of employment, rather than performance.

Integrity/Diversity: Employees ability to communicate openly and freely with management without risk.

Hourly employees appear to feel that staffing decisions are based on the "good ol' boy" system, diverse perspectives are not valued, and that they cannot report unethical practices and bad news to management for fear of reprisal or apathy. This is understandable, considering the historically adversarial relationship between unions and the companies employing their workers.

Salaried Employees

Overall: (or "Job Quality") Future employment with the company, wage levels, improvements made addressing survey responses and comparison of the company to others.

This is not surprising, given that at the time of the survey, there was a unionization effort underway for the salaried employees, based largely on perceived inequities between hourly and salaried benefits packages. Since the time the survey was administered, a majority of salaried employees has in fact unionized, and it is interesting that this may have been prevented had the company realized how strong the feelings of the salaried employees were as indicated from the survey results.

It would be reasonable to assume that differences between hourly and salaried employees may be due to differences in education level, since most hourly employees tend to have fewer years of post high school education. However anecdotally, there were similarities found between hourly employees and the already unionized, salaried engineers, suggesting that unionization may be the underlying variable. It will be interesting to see whether the responses of newly unionized Salaried employees on future Opinion Surveys will in turn begin to resemble those of the Hourly employees.

CONCLUSIONS

In closing, while a non-significant finding is often considered disappointing, in this circumstance it should be seen as a benefit for two main reasons:

• This survey tool, used within this environment and with this population yields comparable responses regardless of the medium. This enables researchers to use either web or paper survey dissemination method interchangeably, according to the needs of the participant population. Caution should be used, however, in generalizing the results of this study to other surveys. Corporations interested in using the Web as the tool for dissemination should conduct their own comparative studies to ensure equivalence.
  
  
• Based on a review of the literature, which reports differences between responses to web and paper surveys, it appears that the determining factor for differentiation is the degree of threat perceived by the participant. This could mean that employees perceive the web and paper modes of the survey as equally non-threatening. It is not known how these results may differ under different conditions, such as times of layoffs.

1Brian Laughlin is a Ph.D. graduate from Wichita State University and is now working as an e-Business Strategist/Developer.