Evaluation: Never More Important

This first Information Resources column deals with evaluation — the foremost consideration in using health-related resources on the Web. Librarians and nursing faculty have a long history of teaching students that not all published information is of the same quality or credibility and what criteria need to be used for making such determinations, such as the differences between a periodical and a scholarly journal. Now, in the networked environment, evaluation has become more challenging. Anyone with the will and some computer know-how can become their own publisher on the Web. Not only does this by-pass the traditional filters for quality that we have come to rely on for the traditional journal literature (e.g., journal reputation, refereed review, credentialed authors and editors), but the electronic medium seems to serve up information with a certain mystique that encourages many to accept what is offered without question. It also tends to obscure some of the distinguishing characteristics of resources that are often more obvious with print products.

Without a healthy skepticism and set of evaluation criteria, the health consumer on the Web can be taken in by fraudulent claims and inaccurate information. Events have shown that the Web is a profitable haven for those promoting bogus and potentially harmful cures, drugs, and devices. The Federal Trade Commission, in cooperation with the Canadian and Mexican governments, held a "Health Claims Surf Day" to ferret out dubious sites related to six diseases: AIDS, arthritis, cancer, diabetes, heart disease, and multiple sclerosis. Promoters and advertisers providing insufficient documentation were then notified that claims must be supported with reliable scientific evidence (Skolnick, 1997a). The World Health Organization established an ad hoc working group in late 1997 to develop recommendations to help regulate health ads and sale of medical products, but the problems of these being adopted and enforced within a global computer network are daunting (Skolnick, 1997b). The British Medical Journal recently reported that enforcement agencies from 20 countries searched the Internet for "potentially misleading health claims and miracle cures" and also identified numerous bogus claims (Brooks, 1998).

In addition to the problem of fraudulent claims is the danger of inaccurate or incomplete information from seemingly reliable sources. In one recent study, researchers surveyed the Web for parent-oriented health sites that provided information on the management of fever in a child (Impicciatore, Pandolfini, Casella, Bonati, 1997). Forty-one Web sites were identified — 32 were commercial and the remainder from individual practitioners, clinics, and educational organizations. The information given at each site was compared with established treatment guidelines. Only one-tenth of the sites closely adhered to the guidelines. Several gave potentially harmful information (e.g., treating with aspirin). Many failed to provide complete information (e.g., how to take a temperature).

A similar, well-publicized study on the extent to which recommendations found on the Web for the treatment of childhood diarrhea concurred with guidelines from the American Academy of Pediatrics (AAP) found significant misinformation as well (McClung, Murray, Heitlinger, 1998). Only 20% of the 60 traditional medical sites concurred with AAP guidelines. The authors concluded steps to rectify this situation and insure dissemination of accurate information must be taken by physicians, teaching institutions, professional societies, and patients themselves. With regard to the latter, the authors stated "Practitioners need to warn their patients about the need for a very critical review of all medical information obtained from the Web, even when it seems to be from a 'reliable' sources" (McClung et al, para.17).

Web Evaluation Sites

Evaluation, then, is very thorny, certainly for consumers. We can expect that in the future more organized, formal evaluation mechanisms will be put in place by the healthcare and library communities. At present, some Web directories provide "ratings" for sites, but a recent study of these found that less than one-third published their criteria (Jadad & Gagliardi, 1998). Of those that did, there was no indication of formalized evaluation addressing issues of interrater reliability or construct validity. Several of the more notable Web directories with a health-specific scope that you may wish to visit are:

• American Medical Association Library Choices (www.ama-assn.org/med_link/med_link.htm)
• Health on the Net Foundation (www.hon.ch)
• OncoLink's Editors' Choice Awards (http://oncolink.upenn.edu/ed_choice/)
• Physician's Choice (www.mdchoice.com/)
• Six Senses Seal of Approval (www.sixsenses.com/)

Additionally, several sites exist that seek to identify problem Web health sites. These include:

• National Council Against Health Fraud (www.ncahf.org): "... a non-profit, tax-exempt voluntary health agency that focuses its attention upon health fraud, misinformation and quackery as public health problems. It is private, nonpolitical and nonsectarian. The organization is comprised of health professionals, educators, researchers, attorneys and concerned citizens. Its officers and board members serve without compensation." Includes "Recommended AntiQuackery Web Sites."
• QuackWatch (www.quackwatch.com/): "... a member of Consumer Federation of America, is a nonprofit corporation whose purpose is to combat health-related frauds, myths, fads, and fallacies. ... Our activities include: Investigating questionable claims, Answering inquiries, Distributing reliable publications, Reporting illegal marketing, Improving the quality of health information on the Internet, Attacking misleading advertising on the Internet." Includes meritorious and "nonrecommended" sites, as well as "Signs of a 'Quacky' Web Site."

Evaluation Criteria

Beyond the guideposts offered by such Web directories, individual searchers-health consumers and professionals--need to be prepared to make personal judgments about the potential usefulness and reliability of information found on the Web. A set of common evaluative criteria is emerging from both the healthcare and library communities (Silberg, Lundberg, Musacchio, 1997; Smith, A., 1998). The discriminating Web user should ask the following:

1. Who created the site?

• what is their authority?
• do they have expertise or experience with the topic?
• what are their credentials, institutional affiliation?
• is an e-mail address provided for a contact person with responsibility for given content or the site?
• is is the site organization logical and easy to maneuver?
• does the URL suggest a reputable affiliation with regard to the topic — personal or official site; type of Internet domain (i.e., .edu: educational institution; .org: non-profit organization; .com: commercial enterprise; .net: Internet Service Provider; .gov: governmental body; .mil: military body)?

2. Is the purpose and intention of the site clear, including any bias or particular viewpoint?

• are the purpose and scope stated?
• who is the intended audience?
• is the information clearly presented as being factual or opinion, primary or secondary in origin?
• what criteria are used for inclusion of the information?
• is any sponsorship or underwriting fully disclosed?

3. Is the information presented accurate?

• are the facts documented or well-researched?
• do the facts compare to related print or other online sources?
• are the Web resources for which links are provided quality sites?

4. Is the information current?

• is the content current?
• are the pages date-stamped with last update?

5. Is the site well-designed and stable?

• is the site organization logical and easy to manuever?
• is the content readable by the intended audience?
• has attention been paid to presenting the information as error-free (e.g., spelling, punctuation) as possible?
• is there a readily identifiable link back to the institutional or organizational home page?
• is the site readable by those with basic access capabilities (i.e., browsers and network speed)?
• are links provided when necessary to download needed browser plug-ins?
• is the site reliably accessible?

Applying the Criteria

Many sites will fail to address all criteria satisfactorily. The searcher must approach each site with skepticism and the intention of using the criteria to gather evidence of quality or lack of the same. These findings should then be substantiated by efforts to contradict or confirm based on other sources.

The Bottom Line

Finally, the power of the health consumer movement is to encourage individuals to be informed about best health practices and treatment options so that they are better prepared to make decisions regarding their personal health. The Web offers exciting opportunities for individuals to seek information appropriate to personal needs. It is the job of health professionals and librarians to underscore with individuals that what is learned on the Web must complement the communication that takes place with their healthcare provider.

THE AUTHOR

Barbara F. Schloman, PhD, AHIP
Director, Library Information Services
Libraries & Media Services
Kent State University
Kent, OH 44242
E-mail Address: schloman@kent.edu