The last Information Resources column on evaluating health resources, "Whom do you trust? Evaluating Internet health resources," was published in January 1999. Since then, there has been considerably more attention paid to the quality of health information on the Web. This is a review of some of the research on Web information quality, studies evaluating that research, third-party rating initiatives, and questions raised about the need to do anything. Educating health consumers to be critical in their choice of health information remains a key strategy—one that health professionals are well-positioned to advance.

Consumer Use of Health Information on the Web

There is strong evidence that consumers are actively seeking health information on the Web. The most detailed information comes from a recent report from the Pew Internet & American Life Project (May 22, 2002). The report focuses on user behavior—how users search for health information and what they choose to believe. Findings from this Pew study include:

• The number of American adults going to the Web for health information in 2002 was 73 million, up from 52 million in November 2000.
• The user group included more women than men (72% compared to 51%). 71% of those 50 to 64 years of age were "health seekers," compared with only 53% of those between 18 and 29. More education and more Internet experience were also predictors. The study did not find any differences between ethnic groups.
• In a typical day, 6 million Americans seek health information on the Web--more than visit a health professional.
• The health seekers surveyed did not have a search plan to find information and started with a search engine or directory, rather than a health Web site.
• Only one-quarter was diligent in evaluating Web sites using accepted criteria (checking currency and source, validating findings).
• Health seekers did report caution in using the information found to make decisions regarding health.
• Of respondents, 61% reported the Internet has improved how they care for their health.

An earlier report from the Pew Internet & American Life Project published in 2000 found the Web was a highly popular source for health information because of its 24-hour availability, assumed anonymity for the health seeker, and abundance of information available. At that time, 30% reported they typically checked out four or more Web sites. While 82% indicated concern about obtaining unreliable information, 52% of those who used health sites believed that "almost all" or "most" information seen on the Internet was credible. Those under 40 years and with less formal education were more accepting of the legitimacy of the health information they retrieved. For this study, 70% indicated the information influenced their health decisions.

The Status of Quality of Health Information on the Web

What the research shows

There is a developing body of research dealing with the Internet as a vehicle for communicating health information. This includes studies centering on quality issues. Some of these have looked at the quality of Web sites on specific health topics and reported areas of concern. The studies published from 2000 to date include: breast cancer (Meric et al, 2002), cancer (Biermann, Golladay, Greenfield, & Baker, 1999), carpal tunnel syndrome (Beredjiklian, Bozentka, Steinberg, & Bernstein, 2000), depression (Griffiths & Christensen, 2000), melanoma (Bichakjian, Schwartz, Wang, Hall, Johnson, & Biermann, 2002), orthodonture (Jiang, 2000), and pharmaceutical company Websites (Griffiths, Christensen, & Evans, 2002).

Other studies analyzed what criteria relate to quality information. Fallis & Frické (2002) identified Web pages on treating fever in children and analyzed what indicators on those Web pages correlated with accuracy. Indicators with a demonstrated relationship were: displaying the Health on the Net Foundation logo (HONcode), having an organization domain (.org), and displaying copyright protection. Display of the HONcode logo was almost four times more likely to be displayed on a more accurate than on a less accurate site. Kunst, Groot, Latthe, Latthe, & Khan (2002) evaluated whether Web sites considered to be credible--because they met the criteria of identifying information source, indicating currency, and presenting hierarchy of evidence--did in fact provide accurate information. Their results found that sites likely to be judged credible based on these criteria correlated only slightly or at best moderately with accuracy.

Two other studies of particular interest looked at quality more from the user experience. Berland et al (2001) searched English- and Spanish-language search engines for information on four common health conditions. Given that health consumers rely on search engines to lead them to health information on a given topic, this study underscored problems with that approach. Findings showed that search engines were only "moderately efficient" in providing sites with relevant content on their first-page of results. These lead sites were then reviewed by expert panels to judge level of coverage and accuracy of content. They found only half of the topics deemed important for a given health issue were covered more than minimally, with even less coverage on the Spanish-language sites. Accuracy, where the text was completely correct for covered clinical elements, varied from 53% to 91% across topics and language of sites. In addition, conflicting information was found in over half of the English-language sites; no conflicts were identified in the Spanish-language sites.

Another user-centered study (Eysenbach & Köhler, 2002) used qualitative means to investigate health consumer Internet search techniques and the criteria used for assessing Web site credibility. Although participants displayed "suboptimal" search techniques, they were successful on average in finding needed answers to questions in less than six minutes. There was no correlation between Internet experience and search time. Focus group participants indicated the assessment criteria they used included: source, professional design, scientific or professional touch, and ease of use. However, the participants in the observational study made little use of source information. They did not probe for information about who was responsible for a site, nor in the post-search interviews could they identify the sites from which they had retrieved information. The authors concluded more work is needed to design both educational strategies and technological means to guide health consumers to quality information.

What the research about the research shows

Eysenbach, Powell, Kuss, & Sa (2002) examined empirical studies on quality of health Web sites. Their objectives included determining what criteria are used, assessing the methodological rigor of the studies, and suggesting future directions. 79 distinct studies were evaluated. The most frequently used quality criteria in these studies were accuracy, completeness, readability, design, disclosures, and provision of references. 70% of the studies determined that quality is a problem. As a group, these studies were lacking in the degree of rigor employed with their evaluation methodology. The authors concluded that research on Web quality would be strengthened by the development of operational definitions for quality criteria.

Two recent studies have explored whether health information on the Internet is linked to cases of harm for health consumers. Bessel, McDonald, Silagy, Anderson, Hiller, & Sansom (2002) identified ten studies that evaluated the use of the Internet to deliver health programs, interventions, and services as compared with other approaches and the influence on decision-making, attitudes, knowledge, satisfaction, health outcomes, and utilization. This study also concluded that there is a decided lack of rigorous research on the relationship between consumer use of health information on the Internet and health outcomes. They found some evidence to suggest that health consumers may in fact obtain needed information in a timely manner to achieve positive health outcomes.

Crocco, Villasis-Keever, & Jadad (2002) also systematically reviewed the literature to identify any reported cases of harm emanating from use of health information obtained on the Internet. They identified risks as due to use of irrelevant or inaccurate information or to the misunderstanding of relevant and valid information. Three articles met the study’s inclusion criteria. One article identified two cases of emotional distress resulting from inappropriate search strategies on the Internet that led to irrelevant information. Two articles reported physical harm. One was the result of a dog owner using erroneous information to treat his three dogs. All recovered following intensive care. The other article related to a man using an unregulated alternative for cancer therapy that is marketed on the Internet. The patient did die. The study authors suggest this relative dearth of reports of harm may be due to actual low risk, underreporting of cases, or bias.

Third-Party Efforts to Insure Quality

"Long-standing" Efforts

Several initiatives to improve quality on the Internet, both by recognizing quality and warning of fraud, have been in place for some time.

HON Code of Conduct (HONcode)

www.hon.ch/HONcode/

The Health on the Net Foundation Code of Conduct was launched in 1996 as a means to standardize requirements to establish the reliability and credibility of health information. "It is a self-regulatory, voluntary certification system based on an ‘active seal’ concept." To obtain certification, a Web site applies for registration. If accepted, the site agrees to abide by the HON principles and qualifies to display the HONcode seal. HON does random checks of sites to check for compliance, as well as relying on reports from the public.

Quackwatch: Your Guide to Health Fraud, Quackery, and Intelligent Decisions

www.quackwatch.org/

The Quackwatch site was established in 1996. "Quackwatch, Inc., a member of Consumer Federation of America, is a nonprofit corporation whose purpose is to combat health-related frauds, myths, fads, and fallacies. Its primary focus is on quackery-related information that is difficult or impossible to get elsewhere." Using a worldwide network of volunteers and expert advisors, Quackwatch investigates questionable claims, reports illegal marketing, generates consumer-protection lawsuits, and undertakes research projects (e.g., alternative cancer treatment registry, dubious advertising, quackery for pets).

U.S. Federal Trade Commission. Operation Cure.All

www.ftc.gov/cureall

"The FTC works for the consumer to prevent fraudulent, deceptive and unfair business practices in the marketplace and to provide information to help consumers spot, stop and avoid them." This site provides information on consumer issues and on how to file a complaint. To combat health fraud on the Internet, the FTC launched Operation Cure.All in 1999. It is an ongoing federal and state law enforcement and consumer education campaign.

More Recent Efforts

Risk (2001) provides a thorough review of the self-regulatory initiatives being undertaken in the English-speaking world. Each initiative promotes a set of criteria with the common elements of "honesty, privacy, confidentiality, accuracy, currency, disclosure, and accountability." Organizations implement their criteria through self-certification of adherence by participating Web sites, third-party certification, or tool-based evaluation. Some of the more recent efforts highlighted by Risk include:

• eHealth Code of Ethics (http://ihealthcoalition.org): Developed in 2000 by the Internet Healthcare Coalition a not-for-profit organization. The Code provides ethical principles for the development of quality standards.
  
  
• European Commission Quality Criteria for Health Related Websites (http://europa.eu.int/information_society/eeurope/ehealth/quality/): Begun in 2001, this effort relies on voluntary support of the quality criteria by member states of the European Commission.
  
  
• Health Internet Ethics (http://www.hiethics.com): Hi-Ethics Inc. is a "not-for-profit consortium of U.S.-based commercial health Internet companies" that began in 2000. Commercial Web sites pay a membership fee and receive certification by following Hi-Ethics principles.
  
  
• MedCERTAIN (www.medcertain.org): Demonstration project of the European Union begun in 2000. Participating Web sites follow metadata labeling protocols and receive a third-party rating and award of a trust mark.
  
  
• URAC Health Web Site Accreditation Programme (www.urac.org/webbsiteaccreditation.htm): URAC was formerly the American Accreditation Healthcare Commission. It offers ten different accreditation programs for managed care organizations and began the program for health Web site accreditation in 2001. URAC accreditation of health Web site organizations includes a fee-based application process with a review to insure the Web site meets quality standards.

Problems with Third-Party Quality Ratings

These various rating systems have limitations as to how effective they can really be. As Risk (2001) suggests, for programs to be meaningful health consumers need to be aware of the quality issues, relate to rating systems, and understand their meaning. Web site providers must commit to implementing and maintaining quality standards. The rating organizations themselves are vulnerable because of the resources required to sustain their programs. Also, the quality ratings have no enforcement power. Losing accreditation would be a modest sanction for any Web site and would not necessarily impede its ongoing existence. Finally, given the voluntary nature of these programs and the number and variety of health sites on the Internet, it is difficult to imagine any of the rating systems taking hold and establishing a pervasive presence.

Several authors have challenged the legitimacy of rating systems. Gagliardi and Jadad (2002) updated a study first done in 1998 (Jadad & Gagliardi) that analyzed the extent to which developers of rating systems had addressed issues of reliability and validity. Both studies found that most systems provided no information on how their evaluation criteria were determined. Of the few that did, there was no evidence that the instruments were validated. Many of the rating organizations in place in 1998 no longer existed by 2002, although health Web sites continued to display their seal. These authors raise the questions of the sustainability of these rating schemas and whether it is feasible to assess quality of health information lacking a gold standard for what defines that construct.

Rating Systems the Way to Go?

Possibly one of the most thought-provoking questions comes from Eysenbach, Powell, Kuss, & Sa (2002) who question whether the Internet really poses problems with regard to accuracy of health information that are unique from other media and that warrant special surveillance. They cite a variety of studies that identify inaccurate or incomplete information in broadcast and print media and suggest that the quality of information on the Internet should be interpreted in the larger context of health information provided through all channels. Delamothe (2000) also questioned the feasibility of trying to assure quality on the Internet, as well as whether such quality control efforts are really needed. He maintains that people will gravitate to those sites that give them what they want and that market forces and "brand loyalty" will do a great deal to overcome any problems.

Conclusion

So after several more years of experience with health information on the Web, what do we know? First, it is clear that more people are becoming regular users of the Web and that health information is a very common target of their searching. Although these health information seekers often do not use preferred searching protocols or evaluation methods, they seemingly are applying a measure of caution in using the information identified. Although the research on Web health information quality has identified inaccuracies and incompleteness, that same body of research has been criticized for its lack of methodological rigor. And in fact there is no significant documented evidence of harm arising from health information provided on the Web.

Nonetheless, efforts to identify quality Web sites continue, but are not without their problems. Third-party rating systems have been difficult to sustain and have gained only a modest acceptance among health sites. Research scrutinizing these has found that the evaluation instruments used by these rating organizations have not been validated, calling into question the legitimacy of the standard applied. Finally, we are faced with the question as to whether problems with health information on the Web are really so different from those presented in print or broadcast media and necessitate extraordinary efforts.

At this moment in time, we are still experiencing tremendous growth and flux with the Internet as a medium for health information. Because of the vastness of this information landscape and the immediacy of access to it—both for creators and users--a strong argument can be made that it is different from other communication media. Its potential for influencing the decisions consumers make regarding their health make the issue of quality important to pursue.

In another four years’ time, we can imagine more efforts to identify quality sites—either by more finely developed rating systems or through the use of Web software tools—will have surfaced. In the meantime, health professionals are positioned to play an important role in educating their clients to be critical users of the information they find on the Web. These guidelines still apply:

1. Who created the site?
   • Authority, credentials, institutional affiliation
2. Is the purpose and intention of the site clear, including any bias or particular viewpoint?
   • Intended audience, purpose/scope, disclosure of sponsorship or underwriting, privacy statement
3. Is the information presented accurate?
   • Facts documented and comparable with other sources; links to quality sites
4. Is the information current?
   • Pages date-stamped, other evidence of updating
5. Is the site well-designed and stable?
   • Logical organization, easy to maneuver, identifiable link to organizational home page, reliably accessible

Also, remember the value of recommending a user start with a health Web directory that has links chosen for their credibility and usefulness, rather than with a generic search engine. Examples of such Web directories would be Healthfinder (www.healthfinder.gov) and MEDLINEplus (www.nlm.nih.gov/medlineplus/).

Let’s see what the next four years bring!

Author

Barbara F. Schloman, PhD, AHIP
Assistant Dean, Library Information Services
Libraries & Media Services
Kent State University
Kent, OH 44242
Email Address: schloman@kent.edu

Disclaimer: Mention of a Web site does not imply endorsement by the author, OJIN, or NursingWorld. Every effort is made to insure currency of Web links at time of publication only.