Also available in [PDF] format.

Today, everybody knows that the Internet offers information and data from many sources worldwide. Yet, many people do not know how to search the Internet most effectively, and many do not know how to evaluate the information they retrieve. Simply because so much information is available—from entertainment to news and from propaganda to opinions to serious research—users need special skills to find and evaluate Internet information. This document guides the reader through six steps to searching for and evaluating information on the Internet. The document uses the following case study as an example: “I work at the XY State Teen Pregnancy Prevention Council. My Executive Director asked me to find information about successfully implementing science-based, teen pregnancy prevention programs in local communities in the United States.”

Step 1: Understand and State the Need.

Don’t just jump right in and type a few words in Google. Take time before starting your search to clearly identify the user’s (or your own) information needs. Be clear about: what is and is not needed, how much information, of what quality, and by when. This means asking questions. In the case study, you might ask the Executive Director (ED):

• Should the information be only about teen pregnancy prevention programs or, for example, can it be about smoking prevention or violence prevention programs?
• Is it important that the programs are ‘local’ or could programs be regional or statewide?
• What does the ED mean by science-based? Evaluated programs with published evidence of effectiveness? Theory-based programs? Something else?
• What type of information does the ED want? ‘How-to’ information on implementing such programs? Information on where to purchase or get technical assistance on implementing such programs? Original, published research about programs’ effectiveness? Research summaries or analyses? Curricula? News summaries?
• How much information does the ED want? For example, everything available on teen pregnancy prevention or one or two research articles?
• Is currency important? That is, does the ED want only recent articles (published in the last three years, for example) or the best research available, regardless of how old it is?
• How will be ED use the information? How soon does the ED want it?
• In what format does the ED want the information?

Step 2: Develop a Search Worksheet:

Before you begin to search the Internet, develop a worksheet on the question you are researching. Include on this worksheet any specific search requirements you must meet. You will probably want to update the worksheet after you perform a preliminary search. A sample worksheet follows:

Search Requirements Worksheet

USER: Executive Director at the XY State Teen Pregnancy Prevention Coalition, an expert on the topic of teen pregnancy prevention

TOPIC (the question): Research on evaluated, effective, science-based teen pregnancy prevention programs in local U.S. communities

USER’S PURPOSE: The information will be used to convince funding agencies to support the coalition in promoting science-based programs in local communities. The information is needed for a grant proposal due in five days (high priority).

SUBJECT AREAS: Adolescent health; sex education; health education; pregnancy prevention; health promotion

ANSWER FORMAT: Two to three highly pertinent research documents, especially peer reviewed journal articles, printed in their entirety along with one-page summaries

SEARCH REQUIREMENTS: High precision and low recall preferred [‘Precision’ means how closely the information matches the user’s question; ‘recall’ refers to the quantity of documents retrieved. Here, we are talking about finding a very few documents that are precisely on target to meet the ED’s request.]

CRITERIA FOR ENDING THE SEARCH: When not more than five, highly relevant, peer-reviewed documents are found; needed by the day after tomorrow

Step 3: Develop the Search Strategy.

You develop the search strategy in three, sequential actions.

First: Conceptualize Your Query.

To develop a successful search strategy, you need to conceptualize the query. [This means thinking through how you will ask the question on various search engines and/or databases.] Consider what you want (for example, documents, images, or videos). Think out both explicit and implicit concepts (and their synonyms) for your search. Pregnancy prevention is an explicit concept of this search while contraceptive education and sex education are implicit concepts. Your query might look like this:

SEARCH FOR: Documents

SELECT: Peer reviewed journal articles

TERMS/SUBJECTS/CONCEPTS: (Science-based practices OR Evidence-based programs OR Programs that work OR Model programs OR Scientific standards OR State-of-the-art) AND (Teen pregnancy OR Teenage pregnancy OR Adolescent pregnancy OR Sex education OR Contraceptive education) AND (Local implementation OR Community-based) AND Evaluation

LIST/SHOW: Title, Author, Journal, Publication date, Full text

Second: Select and Sequence Your Sources.

If you do not know much about your topic, it may be most useful to begin your search in the “visible Web,” using a Web directory (such as Yahoo! Search Directory, http://dir.yahoo.com/) or a Web search engine (such as Google, http://www.google.com/, or Scirus, http://www.scirus.com/). These may assist you in identifying appropriate terminology for your search as well as prominent and reputable publishers, organizations, and individuals working in the field you are researching.

If you are an expert on the topic you are researching, consider beginning your search on the Web sites of expert organizations. In this case study, for example, you could go directly to the Web sites of Advocates for Youth; Alan Guttmacher Institute; Centers for Disease Control and Prevention, Division of Reproductive Health; Child Trends; ETR Associations; National Campaign to Prevent Teen Pregnancy; and/or Sociometrics Corporation.

Or you could begin your search on the “invisible Web,” the content of specialized, Web-based, searchable databases. Search results from these databases are delivered in dynamically generated Web pages and are, therefore, not reached via search engines. [Think of this as the difference between what is visible to the naked eye and what can be seen only with the assistance of a microscope; or consider that, when you go into a searchable database, it is like being on a treasure hunt. You can find some things on the surface, simply by looking. But other treasures can only be found with a map. The “invisible Web” is the map.]

Some searchable databases of particular value to those searching for accurate, research-based, U.S. teen pregnancy information on the Internet include:

• Advocates for Youth’s JournalNet and BiblioNet (coverage of adolescent sexual health research literature)—http://www.advocatesforyouth.info/dbtw-wpd/textbase/afyj.htm and http://www.advocatesforyouth.info/dbtw-wpd/textbase/afyc.htm, respectively
• National Library of Medicine’s and National Institutes of Health’s MEDLINEplus and MEDLINEplus Español (coverage of consumer health information)—http://medlineplus.gov/ and http://medlineplus.gov/spanish/, respectively
• National Library of Medicine’s PubMed (coverage of medical literature)—http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed
• U.S. CDC NPIN Educational Materials Database (coverage of HIV/AIDS, STD, and TB prevention educational materials)—http://www.cdcnpin.org/scripts/locates/LocateMatl.asp?SearchType=Advanced
• U.S. Department of Education’s ERIC (coverage of educational literature)—http://www.eric.ed.gov/

To find other Web-based searchable databases, browse a Web directory or search for a subject term and the word “database” in a Web search engine (e.g., teen pregnancy database or HIV database or health database). In the end, make a short list of the searchable databases, directories, and/or search engines you will use, beginning with those most likely to bring you the desired results.

Third: Translate the Conceptual Query Formulation.

Read the “help, “tips,” or “how-to” page for each chosen search directory (such as the Yahoo! Help Directory http://help.yahoo.com/help/us/dir/), search engine (such as the Scirus Search Tips, http://www.scirus.com/srsapp/tips/), and Web-based searchable database (such as Advocates for Youth’s How to Search Online, http://www.advocatesforyouth.org/factsfigures/help.htm). These pages will help you figure out how best to execute your search. Read these pages often if you use the directory, engine, or database frequently because frequent upgrades allow more powerful searching.

Tips

• Most search systems allow truncation. This is a great asset because truncating words can simplify and improve your query. For example, teen* pregnan* will find at least six phrases: teen pregnancy OR teenage pregnancy OR teens’ pregnancy OR teen pregnancies OR teenage pregnancies OR teens’ pregnancies.
• Most search systems also allow Boolean and proximity searching (combining words and phrases with AND, OR, NOT, p#, and w#). See examples below and remember: read the help page in each search system because the symbols and rules vary for each.
• In most search systems, words joined by AND or by OR are evaluated in left-to-right order: red AND white OR blue finds items that are red and white or it finds items that are blue. To find what you really want, use parentheses to control evaluation order: red AND (white OR blue) finds items that are red and white or red and blue.
• In some databases, enclosing a phrase in quotation marks will find that exact phrase rather than the individual words.

Step 4: Execute Your Search Strategy.

Execute your search in at least two different search systems. Results will vary greatly across search directories, search engines, and Web-based searchable databases. Earlier, you selected and sequenced your sources. Now, it is time to execute the search in the order you selected. Don’t get discouraged if at first you don’t find the results you are looking for. Review at least the first 100 results retrieved by a search engine. You might be surprised at how many times that perfect piece of information is found, not on the first, but on the eighth or later page of results! The engine tries to sort by relevance, but remember that its criteria for relevance aren’t the same as yours.

Don’t forget that searching for information on the Internet is a repetitive process. Don’t get bogged down in a strategy that doesn’t work. If you have executed your search in all the selected sources without positive results, start over. Rethink your query, retranslate your query into a searchable question, and re-execute the revised query in the same search engines and in the same order. If you get frustrated, take a break. But don’t give up; the information most likely does exist—you just have to rethink how to find it. For example, this controlled vocabulary* search—(adolescent pregnancy OR pregnancy prevention) AND (intervention evaluation OR program assessment) AND USA—will get highly pertinent results for your query in Advocates for Youth’s JournalNet. But you have to use the controlled vocabulary for best results. And you will need to modify the query to find relevant results in, for example, the U.S. National Library of Medicine’s PubMed.

Step 5: Evaluate Your Search Results.

Not everything you find on the Internet is good information. Points to consider in evaluating any information resource, print or electronic, include:

• Authorship—This is a major criterion for evaluating information. Ask yourself: Are the authors well-known in the discipline? What are the authors’ qualifications? Do the authors’ prior education, work, and publications relate to the topic under research?
• Publisher—This is an equally important criterion, but it is one that people often neglect. Ask yourself: Is the publishing organization clearly identified on the document? If the document is a journal article, is the journal peer-reviewed? Are the reviewers well-known and knowledgeable in the field? If the document is published by an organization, is there a logo indicating that the document has been authorized by an official academic or scholarly organization? Is this organization or publisher recognized as an expert in the field you are researching?
  
  Documents that are part of an official Web site are more likely to have undergone screening to ensure they meet the standards of the organization publishing the document. U.S. government sites end in .gov, .mil, or .us; educational sites end in .edu; nonprofit organizations often end in .org; and commercial Web sites often end in .com. Look for links that say “About us,” “Philosophy,” “Background,” or “Who We Are.” If you cannot find such links, truncate the URL by deleting the end characters of the URL stopping just before each slash. Press enter to see if you can see more about the author or the origins and nature of the site providing the page. Continue this process, one slash (/) at a time, until you reach the first single / which is preceded by the domain name. This is the page’s publisher.

• Objectivity—Since information is almost always used selectively to present and/or to support a point of view, documents are rarely truly neutral. Moreover, many sexual health issues on can be controversial. Ask yourself: Is the material objective and balanced? Is the information presented fairly, with differences of opinion clearly stated? Is there any advertising on the page? If yes, what does the advertising say about the objectivity of the article, its authors, and its publisher?
• Referral to and/or acknowledgment of the literature—Authors and publishers should acknowledge the information’s relationship to seminal and related work in the same discipline. Ask yourself: Does the document clearly identify its sources? Does the document list bibliographies, directories, or sources of additional information? Do the authors discuss the document’s contribution to the discipline and to other documents, theories and/or schools of thought?
• Accuracy or verifiability of details—Anyone can publish information on the Internet; no Web standards exist to ensure the information’s accuracy. Ask yourself: Are the sources of statistics and other data clearly stated? Can these sources verify the accuracy of the data in this document? Is the information reliable and error-free? Did an editor or review board fact check the information? Is an explanation of the research methods included? Does the author’s conclusion correspond to the findings presented here?
• Currency—For public health information, currency is extremely important. For example, one would not want to present 1989 AIDS death rates for a discussion of the current HIV epidemic. On the other hand, currency is not an issue for some fields, such as history. Ask yourself: Does the document refer clearly to the date the information was gathered (for example, “using data from the 1995 National Survey of Family Growth” …”)? Does the introduction clearly indicate the cut-off date for data (for example, this study was performed using data gathered between January and July, 2005)? Does the document include a date of publication, revision, and/or copyright? What is the time lag between the date of publication and the date of the latest citations and/or resources and is this time lag appropriate?**
• Additional Internet-specific criteria include: 1) accessibility (Must you download software to access the information?, ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', '); 2) design (Is it readable?, ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', '); and safety (Is the document in a format that is unlikely to import a virus into your computer?).

Step 6: Provide Search Results to the User.

When presenting the search results to the user (including yourself), return to the original search request. Review the initial search requirements worksheet and be sure that you’ve met the requirements. For example,

USER: Consider whether the user is an expert or layman in the topic area and present results accordingly. In this case study, the end user is an expert in the field and will be able to read and understand scientific publications on evaluation results.

TOPIC: Make sure that the documents presented to the user answer his/her real question. If you cannot find an answer to the user’s question, be sure to consult with the user to modify the search query. For example, if you couldn’t find information on teen pregnancy prevention programs, would information be useful from other science-based programs addressing teenage health issues, such as substance abuse prevention or smoking prevention?

USER’S PURPOSE: Knowing the user’s purpose can provide a timeline for completing the search as well as with implicit requirements. In this case study, the user needed the information in less than five days in order to incorporate it into a grant proposal.

SUBJECT AREAS: Understanding the larger context of the question may help you select and sequence sources. For example, in this case, understanding that the query lies within the larger fields of health and education, may lead you to first search the Web-based searchable databases of PubMed and ERIC, respectively, or to go to JournalNet, where Advocates for Youth offers access to its special library of adolescent sexual health research.

ANSWER FORMAT: If the user asks for newspaper articles, present one or more newspaper articles. If the user asks for peer reviewed journal articles, present exactly that. In this case, the ED asked for the actual articles along with a summary of the research.

SEARCH REQUIREMENTS: In this case, a few excellent documents are far more valuable to the ED than many documents that only marginally fit. In other cases, the user may want low precision and high recall—meaning everything available on the general topic of, say, adolescent pregnancy prevention, without caring whether the material focuses on effective programs.

CRITERIA FOR ENDING THE SEARCH: Information overload is common in today’s multimedia environment. Do not provide someone who wants a single, five-page research article with 100 pages of articles, news clippings, and statistics. Here, the ED requested one to three highly relevant research articles, with a one-page summary of each. Give those, and nothing more, to the ED. (However, you might say, when presenting the information, that you also identified research on several additional programs, if the ED needs more information.]

Conclusion:

Today, the Internet offers an extraordinary amount of information—good, bad, and indifferent. Searching the Internet skillfully will save time and energy in the long run and will assist users in avoiding the traps of propaganda, distortions, theatrics, and skewed data. Even though conducting a skillful search may take more time, the results will be well worth it—giving youth-serving professionals information on the best and most effective programs, strategies, data, and Web sites. Youth-serving professionals need accurate information related to effective programs and other science-based practices, and it is possible to get this information via skillful searches on the Internet.

Additional Resources

For more information on how to find and evaluate information on the Internet, visit:

1. The University of California Berkeley’s Finding Information on the Internet: A Tutorial online at http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/FindInfo.html
2. Johns Hopkins University’s Evaluating Information Found on the Internet online at http://www.library.jhu.edu/researchhelp/general/evaluating/index.html

* Controlled vocabulary is language that is chosen for uniformity and to give best search results. Controlled vocabulary varies by database and is usually found under Subject or Subject Heading.

** Normal time gaps between the gathering of data and the publication of findings related to the data can be as much as three to five years—up to three years or more for the researchers to sift the data, assess, and write up their findings and up to 18 additional months before the article finally appears in a peer-reviewed journal. Peer-reviewed journals normally note (on the article) when it was submitted for publication. At least some of the citations should reflect dates very close to the submission date.

Researched and compiled by Ammie N. Feijoo, MLS, September 2005

This publication is made possible through a Cooperative Agreement (No. U88/CCU322137-02) with the Centers for Disease Control and Prevention (CDC). This publication may be copied, reproduced, distributed, or adapted without permission of the author or of Advocates for Youth, provided that the document is not copied, distributed, or adapted for commercial gain and provided that the author and Advocates for Youth are credited as the source on all copies, reproductions, distributions, and adaptations of the material.