Satisfaction based program evaluation

by J. Robert Rossman, Ph.D.

J. Robert Rossman, Ph.D. is currently an Assistant Professor in the Department of Leisure Studies at the University of Illinois at Urbana-Champaign and serves as the Community Recreation Specialist in the Office of Recreation and Park Resources. Dr. Rossman has consulted with agencies about evaluation procedures, and conducted evaluation workshops and research on evaluation techniques. His leisure pursuits include racquetball, fishing, bicycling, and spending time with his wife and two children.

Evaluating program services is one of the most difficult, ongoing tasks faced by program managers. Two of the most common approaches to program evaluation, i.e., standards and goals, are cumbersome in actual practice and often reach uncertain conclusions about program worth because of changing standards and unclear goals. The lack of a method and instrument for routinely collecting evaluation data on program outcomes is a major void in the field. In 1980 a study¹ was undertaken to develop a program evaluation instrument to fill this void. The instrument developed in the study is well anchored in leisure theory and provides information to program managers about the source and intensity of satisfactions derived by individuals from participating in programs. Study results provide data about the satisfactions derived from participating in community leisure programs and also bring into question some current evaluation practices.

Development of the instrument was guided by a review of leisure and evaluation literature. Measuring participant reported satisfactions with leisure engagement is the currently accepted method for investigating an individual's experiencing of leisure. Participant reported satisfaction with program engagement was therefore selected as the criteria on which the worth of leisure programs would be judged. The concept of leisure satisfaction was integrated with the findings from the evaluation literature review and an evaluation instrument was developed which: (1) uses valid evaluation criteria deduced from currently accepted "state-of-mind" theories of leisure; (2) focuses its purpose on evaluating program outcomes; (3) meets the specific needs of program managers; and (4) uses a process which is widely applicable.

To conduct the study, an initial item pool of leisure satisfaction statements was assembled from other completed works on leisure satisfaction. The most parsimonious set of satisfaction statements was identified through a series of procedures including content analysis, a jury of park and recreation professionals, and two empirical tests of the instrument followed by data reduction with factor analysis. The reliability and validity of the resultant satisfaction subscales was investigated by using Cronbach's alpha, regression and discriminant analysis.

Results

Results of the study indicate there are seven underlying dimensions of leisure satisfaction with participation in community leisure programs. The seven satisfaction dimensions include: Achievement, Physical Fitness, Social Enjoyment, Family Escape, Environment, Autonomy, and Risk. Reported satisfaction with programs on these dimensions was not affected by a participant's age or sex. However, how a program is organized for delivery and the types of activities offered does have some influence on the

Personal Achievement - it even makes fence painting satisfying.

Illinois Parks and Recreation 12 November/December 1981

satisfactions derived from participating. Programs organized as instructional classes, open facilities, leader-led programs, leagues and tournaments, and special events were scored differently on some dimensions. Physical fitness, organized sports, mental and linguistic, art and social activity types also produced differing results on some dimensions. These findings indicate that program managers influence the probability of certain satisfactions being realized by their choice of activity types and the way they organize a program for delivery.

Satisfaction with the community recreation programs investigated in the study was principally accounted for by the Achievement, Physical Fitness, and Social Enjoyment satisfaction dimensions. Although the most salient dimensions for various programs and communities may vary, use of the instrument does provide a mechanism for documenting the source and intensity of satisfactions with program services.

Additional study results bring into question some current evaluation practices. Specifically it was found that current data collection techniques are inappropriate, that an "overall satisfaction" measure is not sufficiently discriminating to be useful, and finally that individuals report being satisfied with programs which are not important to them.

Data collection in the study duplicated current practice of distributing evaluation instruments to participants still in a program during a program session toward the end of program operation. At this point in the programs investigated, almost one-half of the registered participants had either dropped out or did not attend on the evening the evaluation was conducted. Doubt about the satisfaction or "dissatisfaction" level of those individuals not completing the instrument remains. With the currently used data collection procedure, one can not be certain that the results obtained actually represent all those enrolled in a program or just those who were highly satisfied and continued with a program. A sampling of all registered participants toward the end of program operation should be used to collect data until it can be demonstrated that satisfaction scores from those who attend regularly throughout a program and those who attend irregularly or drop-out of a program are significantly correlated. Only then can one be certain that high satisfaction scores for programs are not simply the result of omitting all dissatisfied individuals from the data.

Many agencies currently measure participant satisfaction with programs by using an "overall satisfaction" item. The usefulness of an "overall satisfaction" item was investigated by including it on the instrument used in the study. Results indicate that an "overall satisfaction" item does not produce responses with enough variability to permit discrimination between programs. "Overall satisfaction" received the highest mean score and lowest standard deviation of any item on the instrument. This indicates that individuals were in agreement about their high overall satisfaction with programs but that there was little variability to their responses on the item. However, in order to judge the relative worth of individual programs, valid data on which programs vary must be collected. In this regard, the leisure satisfaction dimensions developed in the study provide more discriminating and therefore more useful evaluation information than an "overall satisfaction" item.

How important participation in a program was to an individual and how important a program was compared to all other leisure pursuits of an individual was also investigated. Results of the study reveal it is possible for individuals to be satisfied with programs they indicate are not important to them. If it is a goal of agencies to provide programs that are satisfying and important to the individuals in them, then evaluation instruments need to include measures of both. Participant reported satisfactions with programs can be put in perspective by understanding the importance of participating in a program to the individuals in them.

Physical Fitness - a major satisfaction of leisure.

This study demonstrated that it is possible to evaluate programs with participant reported satisfaction data. However, the usefulness of satisfaction evaluation data can be increased by implementing the following recommendations. First, the notion of satisfaction is best measured by investigating underlying satisfaction dimensions rather than using a single all-encompassing measure of "overall satisfaction." Second, satisfaction data needs to be placed in perspective by investigating the importance of participating in programs to individual participants. Third, more representative information can be collected by sampling a percentage of all individuals registered for a program rather than surveying only those still attending during later program periods.

¹ This study was partially funded by a grant from the Illinois Park and Recreation Foundation.

Illinois Parks and Recreation 13 November/December 1981