Best Practices Methodology

by Stuart Hanson

© InfoUse, 1998

Lois-Ellin Datta (1993) looked at many federal agencies' "approaches to recognizing promising, good, effective, and exemplary practices." She reports on four approaches taken: (1) empirically based strategic planning; (2) consensus conferences; (3) independent validation of expert opinion; and (4) indirect support, cream rises. Each is described below.

Empirically-Based Strategic Planning. The National Cancer Institute has a particularly strong long-term strategy for "moving ideas from fundamental research to wide-spread practice." They have identified seven sequential stagesÑbasic biomedical research, hypothesis development, methods development, controlled interventions, defined population studies, and nationwide prevention and health service programs. Each has a different focus and answers different questions. Movement to the next stage is contingent on two decisions: that the data are convincing enough to be worth further investment, and that the data are among the most convincing.

Consensus Conferences. About 100 consensus conferences have been held since 1977. At the Consensus Conferences on Medical Practices held by the Office of Medical Applications of the National Institutes of Health, the approach was useful for looking at significant and controversial issues of treatment or procedures where there had been enough high quality empirical evidence to reach some conclusion. They have also been used by the Environmental Protection Agency and the Food and Drug Administration.

Independent Validation of Expert Opinion. The U.S. General Accounting Office is a frequent user of the "expert opinion, independent validation" approach for reports serving three legislative purposes, the analysis of options for new legislation, the evaluation of programs in terms of a range of outcomes, to see what has worked, and for recognizing exemplary practices.

Indirect Support, Cream Rises. Some agencies and many fields base recognition of the best on the receipt of awards for outstanding products and outstanding performance or production. Receiving a grant, or an award by a prestigious professional group are equated by some with exemplary results at the end.

Datta also discusses several vulnerabilities that studies of best practices have: undefined universes, non-comparable goals, identifying reproducible conditions, saturation, and dilution. Undefined boundaries refers to the problem of identifying the pool. The use of a nomination process is usually used to deal with this issue. The issue of non-comparable goals is the old problem of comparing apples and oranges.

"Perhaps the most difficult challenge, methodologically, in identifying promising and best practices," according to the author, "is the difference between what may work superbly well and knowing (a) which factors are necessary and sufficient for results to be achieved elsewhere and (b) which of these factors are reproducible in any form or adaptation."

Some programs and practices that work well in a few sites may not work when extended to serve all persons or an entire region, because the demand for resources is greater than supply. In other cases, results may be less when there is movement away from the original project conditions. These are the vulnerabilities of saturation and dilution.

Datta identifies significant concerns and suggests possible solutions to methodological concerns in exemplary practices studies, which should help to optimize the design of the VR-IL Employment Outcome Partnership Project.

The nomination manual of the Regional Rehabilitation Exchange (1992) sets forth a process which it has continually refined since initiating the process in 1983. Programs could be nominated by the Regional Rehabilitation Exchange (RRX) Peer Review Advisory Council or staff, program site staff, consumers, NIDRR and RSA staff, and others, in three well-defined service area categories. Nominees were asked to determine if they met all prerequisites and, if so, to complete an information request form. RRX staff analyzed data by use of weighted evaluation criteria, approving or eliminating sites for further consideration. Site visits to approved sites were then made to verify and collect data, and observe program operations. Anonymous information summaries and staff recommendations were then submitted to the Peer Review Advisory Council for review and final selection. Eleven "critical factors" are identified and described by questions that provide relevant information. The eleven factors are: program goals and objectives; target population; consumer involvement and satisfaction; documentation; success rate; cost-effectiveness; comprehensiveness; evaluation criteria; staffing patterns; transportability; and innovativeness.

The Florida Mental Health Institute at the University of South Florida (InfoUse, 1996) also developed a process for identifying successful educational programs which can, itself, serve as a model for those seeking to identify exemplary programs. Nominations are sought through a variety of methods. An Advisory Panel comprised of professionals, parents, and consumers provides multiple perspectives and input on the participating programs. Two questionnaires are used; the first to determine if basic criteria are met, the second to obtain more detail and outcome indicator information. Sites returning the second questionnaire become eligible for a site visit; priority is given to those that meet selected criteria. Each site is given an honorarium of $500 as a token of appreciation for their participation. The determination of the final sites is made by staff, with input from the Advisory Panel.

The Regional Rehabilitation Exchange (RRX) (1991) published a catalog that identifies exemplary programs and practices in independent living, job placement, supported employment, and transitional services. Funded by a $225,000 per year grant from NIDRR, the RRX mounted a thorough, though largely regional, nomination, screening, and selection process. (On-site visits to nominated sites were conducted to gather information, for example.) Collaborative efforts between organizations were not a critical factor in the selection process, though VR funding and relationships are often mentioned in the catalog's organizational profiles.

Similar catalogs were prepared by the Rehabilitation Network of New England (1989) and others. NIDRR funded a number of "best practices" information dissemination projects during this period.^{Footnote 1} The emphasis in this and the one above is information about programs, not about any collaboration which may underlie the program's existence. In neither does the nature of the collaboration itself emerge as a critical variable.

The U.S. Department of Education (1966), in a proposed regulation, provides another approach to consider when seeking to identify and distinguish exemplary and promising programs. In their proposal, the eligible entity submits a program abstract of 250 words or less; a description of the salient features of the program; a description of the program's philosophy and history; site information, including demographics; a description of evaluation results; funding and staffing information; and contact information. Expert panels are formed to evaluate programs according to a set of established criteria, in four categories: evidence of success; quality of the program; educational significance; and usefulness to others. Sharon Bobbitt, the Department of Education's contact person for this proposed rule, indicated (E-mail, June 30, 1997), "The next version will be substantially pared down from the original, giving the expert panels a lot of leeway to decide the specific criteria that fall under each of the major categories."

At a higher level of abstraction, Backer (1991) and Rich (1991) consider best practices and other studies in the more encompassing framework of knowledge utilization. Backer points out that the knowledge utilization field now has approximately 10,000 literature citations, and identifies some of the basic assumptions, critical challenges, and strategies that have proved effective in meeting these challenges. Rich's article summarizes much empirical and theoretical work in the area of knowledge utilization, but points out "there appear to be serious conceptual and methodological gaps which need to be filled."

---

Footnotes

¹ See National Association of Rehabilitation Research and Training Centers & National Rehabilitation Information Center. (1986), and National Institute on Disability and Rehabilitation Research. (1988, 1989, 1990, 1992, 1993, & 1995). Back to main text.

---

Bibliography

Backer, T. E. (1991). Knowledge utilization: The third wave. Knowledge: Creation, Diffusion, Utilization, 12 (3, March) 319-337.

Backer, T. E. (1991). Knowledge creation, diffusion, and utilization. Knowledge: Creation, Diffusion, Utilization, 12 (3, March) 225-240.

Bishop, J. H. (1989). Toward more valid evaluations of training programs serving the disadvantaged. Journal of Policy Analysis and Management, 8 (2), 209-28.

Bobbitt, S. (June 30, 1997). Re: Query re status of proposed rules. Email message. Washington, DC: U.S. Department of Education.

British Columbia Premier's Advisory Council for Persons with Disabilities. (1991). Harmonization of services, Report of the community services task team [Discussion paper]. Vancouver, BC, Canada.

Datta, L.-E. (1993). Best wishes and many happy returns: Some federal efforts in recognizing the best. Evaluation and Program Planning, 16, 219-226.

Hilton, J. (1997). Survey of state VR agencies on exemplary practices and collaborative relationships that promote employment between state vocational rehabilitation agencies and independent living centers; and between basic vocational rehabilitation service programs and independent living rehabilitation service programs [Memo to state directors of vocational rehabilitation agencies].

InfoUse. (1996). Study 1: The effects of service systems reform, including school restructuring, on children and youth with serious emotional disturbances (CYSED). NIDRR Program Review. Berkeley, CA.

Johnson, M. D., & Fawcett, S. B. (1988). Quality circles: Enhancing responsiveness of service agencies to consumers. American Rehabilitation (Summer), 20-21.

National Association of Rehabilitation Research and Training Centers & National Rehabilitation Information Center. (1986). 1986 Directory of the rehabilitation research and training centers. Washington, DC: U.S. Department of Education.

National Institute on Disability and Rehabilitation Research. (1988). NIDRR program directory. Washington, DC: U.S. Department of Education.

National Rehabilitation Information Center. (1996). NIDRR program directory. Washington, DC: U.S. Department of Education.

National Rehabilitation Information Center. (1995). NIDRR program directory. Washington, DC: U.S. Department of Education.

National Rehabilitation Information Center. (1993). NIDRR program directory. Washington, DC: U.S. Department of Education.

National Rehabilitation Information Center. (1992). NIDRR program directory. Washington, DC: U.S. Department of Education.

National Rehabilitation Information Center. (1990). NIDRR program directory. Washington, DC: U.S. Department of Education.

National Rehabilitation Information Center. (1989). NIDRR program directory. Washington, DC: U.S. Department of Education.

Parent, W., Kregel, J., Wehman, H., & Metzler, H. (n.d.). The Vocational Integration Index: A guide for rehabilitation professionals, special educators, and consumers. Richmond, VA: Virginia Commonwealth University Rehabilitation Research and Training Center.

Regional Rehabilitation Exchange. (1990). Regional Rehabilitation Exchange: Annual report. Austin, TX: Southwest Educational Development Laboratory.

Regional Rehabilitation Exchange. (1991). RRX catalog of exemplary programs and practices. Austin, TX: Southwest Educational Development Laboratory.

Regional Rehabilitation Exchange. (1992). Manual for completing the independent living services information request form. Austin, TX: Southwest Educational Development Laboratory.

Rehabilitation Network of New England. (1989). Catalog of exemplary programs. Andover, MA: The Network, Inc.

Rogers, E. M., & Shoemaker, F. F. (1971). Comunication of innovations: A cross-cultural approach. (2nd ed.). New York: Free Press.

Rogers, E. M. (1983). The diffusion of innovations. New York: Free Press.

Staniszewski, S. C. (1987). Employment and the chronically mentally ill: A return to health and functioning. Albertson, NY: Human Resources Center.

U.S. Department of Education (1996). Standards for conduct and evaluation of activities carried out by the Office of Educational Research and Improvement (OERI)--Designation of exemplary and promising programs: Proposed rule, Federal Register: June 3, 1996, 61 (107).

---

Comments or Questions? Email us.