Richard P. Phelps

Richard P. Phelps' Home Page

Richard P. Phelps's Web Page
-----------------------------------------------------------------------------------------------------------------------------------------------
Education: PhD: U.Penn (Wharton); Master's: Harvard, Indiana; Bachelor's: Washington (St. Louis)
Postings: Organisation for Economic Co-operation and Development (OECD); ACT; National Evaluation Systems; Westat; U.S. General Accounting Office (GAO); Indiana Education Department; Harvard Business School; Professeur de Mathématiques, Séminaire St. Augustin, Baskouré, Burkina Faso
Awards: Research Fellowships: Educational Testing Service; American Education Finance Association; the National Center for Education Statistics; Morganthau Fellow (Wharton); Kennedy Fellow, Gustav Harris Scholarship (Harvard)
Contact: richardpphelps (at) yahoo (dot) com
----------------------------------------------------------------------------------------------------------------------------------------------------

Links and citations are listed in these categories, respectively, below:
  • The Effect of Testing on Student Achievement, 1910-2010 # Correcting Fallacies # Standardized Testing Primer # Defending Standardized Testing # Kill the Messenger
  • International # Testing # Education Finance # Methodological/Evaluation # Source of Lake Wobegon
  • Not on the Web: (International) # (Testing) # (Education Finance) # (Methodological/Evaluation)

  • Nonpartisan Education Review
  • INCREASE2

  • The Rot Festers: Another National Research Council Report on Testing
  • Comments on the draft Standards for Educational and Psychological Testing
  • Educators Cheating on Tests Is Nothing New: Doing Something about It Would Be
  • The Rocky Score-line of Lake Wobegon

  • Censorship Has Many Fathers: A progeny of excuses for suppressing information that one dislikes

    • ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    The Effect of Testing on Student Achievement, 1910-2010 (International Journal of Testing, January 2012)

    This article summarizes research on the effect of testing on student achievement as found in English-language sources, comprising several hundred studies conducted between 1910 and 2010. Among quantitative studies, mean effect sizes range from a moderate d ≈ 0.55 to a fairly large d ≈ 0.88, depending on the way effects are aggregated or effect sizes are adjusted for study artifacts. Testing with feedback produces the strongest positive effect on achievement. Adding stakes or frequency also strongly and positively affects achievement. Survey studies produce effect sizes above 1.0. Ninety-three percent of qualitative studies analyzed also reported positive effects.

    Acknowledgements: Thanks to three anonymous reviewers and editor Steve Sireci for their excellent suggestions. Thanks also to individuals who aided the research, including: Jeff Allen; Kurt Burkum; Richard Coley; Lori Dockery; John Hattie; George Johanson; Kathy Lynch; Karen McQuillan; Katie Paul; John Poggio; Terris Raiford; Jacqueline Snider; Gang Wang; and Karen Zimmerman. Furthermore, credit goes to those who conducted research reviews on relevant subtopics (in chronological order): Panlasigui (1928); Ross (1942); Kirkland (1971); Proger & Mann (1973); Jones (1974); Bjork (1975); Peckham & Roe (1977); Wildemuth (1977); Jackson & Battiste (1978); Kulik, Kulik, Bangert-Drowns, & Schwalb (1983–1991); Natriello & Dornbusch (1984); Dawson & Dawson (1985); Levine (1985); Resnick & Resnick (1985); Guskey & Gates (1986); Hembree (1987); Crooks (1988); Dempster (1991); Cameron & Pierce (1994); Adams & Chapman (2002); Locke & Latham (2002); Roediger & Karpicke (2006); J. Lee (2007); and Basol & Johanson (2009).

    "Most Viewed" article in The International Journal of Testing.


    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    Correcting Fallacies about Educational and Psychological Testing (American Psychological Association, December 2008)

    Standardized testing bears the twin burden of controversy and complexity and is difficult for many to understand either dispassionately or technically. In response to this reality, Richard P. Phelps and a team of well-noted measurement specialists describe the current state of public debate about testing across fields, explain and refute the primary criticisms of testing, acknowledge the limitations and undesirable consequences of testing, provide suggestions for improving testing practices, and present a vigorous defense of testing as well as a practical vision for its promise and future.
    Those who are charged with translating the science of testing into public information and policy—including administrators, social scientists, test publishers, professors, and journalists who specialize in education and psychology—will find a wealth of valuable information here with which to balance the debate.

    Table of Contents:
    Foreword, Thomas Oakland
    Introduction and Overview
    Logical Fallacies Used to Dismiss the Evidence on Intelligence Testing, Linda S. Gottfredson
    Psychological Diagnostic Testing: Addressing Challenges in Clinical Applications of Testing, Janet F. Carlson & Kurt F. Geisinger
    Educational Achievement Testing: Critiques and Rebuttals, Richard P. Phelps
    College Admission Testing: Myths and Realities in an Age of Admissions Hype, Wayne J. Camara
    Criticisms of Employment Testing: A Commentary, Ernest H. O’Boyle & Michael A. McDaniel
    Mission—Protect the Public: Licensure and Certification Testing in the 21st Century, Stephen G. Sireci & Ron K. Hambleton
    Mistaken Impressions of Large-Scale Cognitive Diagnostic Testing, Jacqueline P. Leighton
    Summary & Discussion, Richard P. Phelps & Linda S. Gottfredson


    ISBN: 9781433803925 or 1433803925
    LB3051.C6386 2009
    Dewey: 371.262 C824

    #18 on Best Sellers in Education list for 2009, Library Journal

    Reviews for Correcting Fallacies about Educational and Psychological Testing
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    Standardized Testing Primer (Peter Lang, 2007)

    The Standardized Testing Primer provides non-specialists with a thorough overview of this controversial and complicated topic. It eschews the statistical details of scaling, scoring, and measurement that are widely available in textbooks and at testing organization Web sites, and instead describes standardized testing’s social and political roles and its practical uses—who tests, when, where, and why. Topics include: an historical background of testing’s practical uses in psychology, education, and the workplace; the varied structures of educational testing programs and systems across countries; the mechanics of test development and quality assurance; and current trends in test development and administration. A glossary and bibliography are also provided. The Standardized Testing Primer is an ideal text for teaching this subject to undergraduate and graduate students.


    Table of Contents:
    Introduction
    Aptitude or Achievement: Two Separate Historical Paths
    The Effects of Testing
    Mechanics of Test Development and Quality Assurance
    Conclusion
    References & Resources

    Find at Peter Lang Publishing, at Amazon, or at Barnes and Noble
    Search: FetchBook.info

    ISBN: 9780820497419 082049741X
    OCLC: 155715191 paper
    LOC: LB3051.P543
    Dewey: 371.26'2-dc22

    Reviews for Standardized Testing Primer

    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    Defending Standardized Testing (Lawrence Erlbaum, 2005)

    The education reform movement of the past two decades has focused on raising academic standards. Some standards advocates attach a testing mechanism to gauge the extent to which high standards are actually accomplished, whereas some critics accuse the push for standards and testing of impeding reform and perpetuating inequality. At the same time, the testing profession has produced advances in the format, accuracy, dependability, and utility of tests. Never before has obtaining such an abundance of accurate and useful information about student learning been possible. Meanwhile, the American public remains steadfast in support of testing to measure student performance and monitor the performance of educational systems.

    Many educational testing experts who acknowledge the benefits of testing also believe that those benefits have been insufficiently articulated. Although much has been written on standardized testing policy, most of the published material has been written by opponents. The contributing authors of this volume are both accomplished researchers and practitioners who are respected and admired worldwide. They bring to the project an abundance of experience working with standardized tests.

    The goal of Defending Standardized Testing is to: describe current standardized testing policies and strategies; explain many of the common criticisms of standardized testing; document the public support for, and the realized benefits of, standardized testing; acknowledge the limitations of, and suggest improvements to, testing practices; provide guidance for structuring testing programs in light of public preference and the No Child Left Behind Act; and present a defense of standardized testing and a practical vision for its promise and future.

    Table of Contents:
    J.J. Fremer, Foreword: The Rest of the Story
    R.P. Phelps, Persistently Positive: Forty Years of Public Opinion on Standardized Testing.
    G.J. Cizek, High-Stakes Testing: Contexts, Characteristics, Critiques, and Consequences.
    R.P. Phelps, The Rich, Robust Research Literature on Testing's Achievement Benefits.
    D. Goodman, R.K. Hambleton, Some Misconceptions About Large-Scale Assessments.
    S.G. Sireci, The Five Most Frequently Unasked Questions About Standardized Testing.
    G.K. Cunningham, Must High-Stakes Mean Low Quality?
    C. Buckendahl, R. Hunt, The Relationship Between the "Rules" and "Law" of Testing.
    L. Crocker, Teaching FOR the Test: How and Why Test Preparation is Appropriate.
    B.S. Plake, Doesn't Everybody Know That 70% Is Passing?
    K.F. Gelsinger, The Testing Industry, Ethnic Minorities and Those with Disabilities.
    D. Rogosa, API Awards and the Orange County Register Margin of Error Folly.
    M.L. Bourque, Leave no Standardized Test Behind.

    ISBN: 0805849114 cloth # 0805849122 paper

    Reviews for Defending Standardized Testing

    Available at your friendly neighborhood virtual bookstore, or through Lawrence Erlbaum Associates: # cloth # paper or eBooks

    Amazon # Barnes&Noble # AbeBooks
    Search: FetchBook.info

    ----------------------------------------------------------------------------------------------------------------------------------------------------------

    Kill the Messenger (Transaction, 1st Printing, 2003 (cloth); 2nd Printing, 2005 (paper))

    Description: In response to public demand, new federal legislation requires testing of most students in the United States in reading and mathematics, for grades three through eight. In much of the country, this new order promotes an Increase in the amount of standardized testing. Many educators, parents, and policymakers who have paid little attention to testing policy issues in the past will now do so. They deserve to have better information on the topic than has generally been available, and Kill the Messenger is intended to fill this gap. Kill the Messenger is perhaps the most thorough and authoritative work in defense of educational testing ever written. Phelps points out that much research conducted by education insiders on the topic is based on ideological preference or profound self-interest. It is not surprising that they arrive at emphatically anti-testing conclusions. He notes that external and high stakes testing in particular attracts a cornucopia of invective. Much, if not most, of this hostile research is passed on to the public by journalists as if it were neutral, objective, and independent. Kill the Messenger describes the current debate, the players, their interests, and their positions. It explains and refutes many of the common criticisms of testing. It describes testing opponents strategies, through case studies of Texas and the SAT. It acknowledges testing's limitations, and suggests how it can be improved. It defends testing by comparing it with its alternatives. And finally, it outlines the consequences of losing the war on standardized testing.

    ISBN Nos. 0765801787 (cloth) # 1412805120 (2nd printing, paper) # 1412805124 (3rd printing, paper)
    Library of Congress call number: LB3051 .P54 2003
    Dewey Decimal System number: 371.26/2

    Association of American Publishers' Summary of Kill the Messenger

    Reviews for Kill the Messenger: The War on Standardized Testing

    Selected as a Profiler's Pick by YBP, Inc. and Academia Magazine from among their CORE 1000 recommended books for academic libraries, 2003/2004. "Reviews the debate over standardized testing requirements and argues in favor of testing."

    Available at your friendly neighborhood virtual bookstore: (or through Transaction Publishers)

    Abe Books # Amazon # Barnes&Noble

    SEARCH: FetchBook.info

    -------------------------------------------------------------------------------------------------------------

    The Source of Lake Wobegon

    John J. Cannell's late 1980's “Lake Wobegon” reports suggested widespread deliberate educator manipulation of norm-referenced standardized test (NRT) administrations and results, resulting in artificial test score gains. The Cannell studies have been referenced in education research since, but as evidence that high stakes (and not cheating or lax security) cause test score inflation. This article examines that research and Cannell's data for evidence that high stakes cause test score inflation. No such evidence is found. Indeed, the evidence indicates that, if anything, the absence of high stakes is associated with artificial test score gains. The variable most highly correlated with test score inflation is general performance on achievement tests, with traditionally low-performing states exhibiting more test score inflation—on low-stakes norm-referenced tests—than traditionally high-performing states, regardless of whether or not a state also maintains a high-stakes testing program. The unsupported high-stakes-cause-test-score-inflation hypothesis seems to derive from the surreptitious substitution of an antiquated definition of the term “high stakes” and a few studies afflicted with left-out-variable bias. The source of test-score inflation is lax test security, regardless the stakes of the assessment.

    The Source of Lake Wobegon [Updated]

    The Source of Lake Wobegon [Updated] (pdf file)

    The Source of Lake Wobegon [slide show]

    The Rocky Score-line of Lake Wobegon

    The Source of Lake Wobegon [original]

    The Source of Lake Wobegon [original, pdf]

    Measuring Up: What Educational Testing Really Tells Us, by Daniel Koretz [book review]

    -------------------------------------------------------------------------------------------------------------

    INTERNATIONAL:

    Phelps, R. P. (2013). The rot spreads worldwide: The OECD: Taken in and taking sides. New Educational Foundations, Winter, 2013.

    Phelps, R. P., Zenisky, A., Hambleton, R. K., & Sireci, S. G. (2010, March). Report to Ofqual: On the reporting of measurement uncertainty and reliability for U.S. educational and licensure tests

    Higher Education: An International Perspective (c)2003, Richard P. Phelps
    ...also available in paper and microfiche from ERIC, no. ED474484, Clearinghouse no. HE035711

    Organisation for Economic Co-operation and Development, Centre for Educational Research and Innovation (2000) Investing in Education: Analysis of the 1999 World Education Indicators, Paris: Author, October.

    Trends in large-scale, external testing outside the United States, Educational Measurement: Issues and Practice (EMIP), V.19, N.1, Spring 2000.

    Are U.S. students the most heavily tested on earth? Educational Measurement: Issues and Practice, V.15, N.3, Fall 1996

    U.S. Department of Education. National Center for Education Statistics. Education in States and Nations, (2nd ed.), NCES 96-160, by Richard P. Phelps, Thomas M. Smith, and Nabeel Alsalam.Washington, DC: 1996.

    U.S. Department of Education. National Center for Education Statistics. International Education Expenditure Comparability Study: Final Report, (Vol. 2). Working Paper #17, by Richard P. Phelps and Shelley Burns. Washington, DC: 1997.


    TESTING:

    Phelps, R. P. (2012, October 2). The school-test publishers complex Education News

    Phelps, R. P. (2012, September 19). The (secret) document that drives standardized testing Education News

    Phelps, R. P. (2013). The rot festers: Another National Research Council report on testing. New Educational Foundations, Summer, 2012.

    Phelps, R. P. (2011, Autumn). Teach to the Test? Wilson Quarterly

    Phelps, R. P. (2011). The Effect of Testing on Achievement: Meta-Analyses and Research Summary, 1910–2010: Source List, Effect Sizes, and References for Quantitative Studies. Nonpartisan Education Review/Resources.

    Phelps, R. P. (2011). The Effect of Testing on Achievement: Meta-Analyses and Research Summary, 1910–2010: Source List, Effect Sizes, and References for Survey Studies. Nonpartisan Education Review/Resources.

    Phelps, R.P. (2011). The Effect of Testing on Achievement: Meta-Analyses and Research Summary, 1910–2010: Source List, Effect Sizes, and References for Qualitative Studies. Nonpartisan Education Review/Resources.

    Phelps, R. P., Zenisky, A., Hambleton, R. K., & Sireci, S. G. (2010, March). Report to Ofqual: On the reporting of measurement uncertainty and reliability for U.S. educational and licensure tests

    The Rocky Score-Line of Lake Wobegon

    The source of Lake Wobegon [updated] #

    The source of Lake Wobegon [updated, pdf]

    The National Research Council’s Testing Expertise

    Measuring Up: What Educational Testing Really Tells Us, by Daniel Koretz [book review]

    Characteristics of an effective testing system

    The source of Lake Wobegon #

    The source of Lake Wobegon [pdf] # See Bas Braams' excellent essay on this topic.

    Educational testing policy: Stuck between two political parties

    Standardized testing, regrettable food, urine therapy, and trichotillomania (Adults Only!)

    Organizations and Individuals to Contact for Information on Testing

    Economic Perspectives on Standardized Testing (powerpoint file)

    High Stakes: Testing for Tracking, Promotion, and Graduation, book review, Educational and Psychological Measurement, Vol.60, No.6, December 2000.

    Phelps, Richard P. (1999) "Education Establishment Bias: A Look at the National Research Council's Critique of Test Utility Studies." The Industrial-Organizational Psychologist, 36(4). Society of Industrial-Organizational Psychologists, April.

    The Demand for Standardized Student Testing, Educational Measurement: Issues and Practice, V.17, N.3, Fall 1998.

    Trends in large-scale, external testing outside the United States, Educational Measurement: Issues and Practice (EMIP), V.19, N.1, Spring 2000.

    Are U.S. students the most heavily tested on earth? Educational Measurement: Issues and Practice, V.15, N.3, Fall 1996

    U.S. General Accounting Office, Student Testing: Current Extent and Expenditures, With Cost Estimates for a National Examination. PEMD-93-8 January 13, 1993


    EDUCATION FINANCE:

    Phelps, R. P. (2006, September 1). Thoroughly Inadequate: The 'School Funding Adequacy' Evasion Nevada Policy Research Institute.

    U.S. General Accounting Office, Student Testing: Current Extent and Expenditures, With Cost Estimates for a National Examination. PEMD-93-8 January 13, 1993

    "A Primer on Privatization," July 2001


    STATISTICAL/METHODOLOGY/EVALUATION:

    Phelps, R. P. (2013). The rot spreads worldwide: The OECD: Taken in and taking sides. New Educational Foundations, Winter, 2013.

    "book review: Raising the Grade: How High School Reform Can Save Our Youth and Our Nation by Bob Wise," Educational Horizons,

    "Dropping the ball on dropouts," Educational Horizons,

    "The dissolution of education knowledge," Educational Horizons,

    Standardized testing, regrettable food, urine therapy, and trichotillomania (Adults Only!)

    "A review of the Manhattan Institute's Graduation Rates in the United States," Practical, Assessment, Research, and Evaluation, v.10, n.15

    U.S. Department of Education. National Center for Education Statistics. Features of Occupational Programs at the Secondary and Postsecondary Education Levels. NCES 2001-018, by Richard P. Phelps, Basmat Parsad, Elizabeth Farris, and Lisa Hudson. Project Officer: Bernard Greene. Washington, DC: 2001.

    U.S. Department of Education. National Center for Education Statistics. State Indicators in Education 1997, NCES 97-376, by Richard P. Phelps, Andrew Cullen, Jack C. Easton, and Clayton M. Best. Project Officer, Claire Geddes. Washington, DC: 1997.

    Censorship has many fathers


    Not on the Web (INTERNATIONAL):

    "Benchmarking to the World's Best in Mathematics: Quality Control in Curriculum and Instruction Among the Top-Performing Countries on the TIMSS," Evaluation Review, August 2001

    Review of Survey Items on Continuing Education and Training, NCES-OECD, Feb. 1998

    "Student Achievement and Adult Literacy," chapter 6, Education at a Glance: OECD Indicators, Centre for Educational Research and Innovation, OECD, 1997

    Description and Status of U.S.-based International Benchmarking Efforts, NCES, 1997

    "International Comparisons of Public Expenditures on Education," in: The Condition of Education, 1998, _1997, _1996, _1995, _1994, NCES

    Survey of Countries' Practices Benchmarking Math Standards, NCES, 1996

    "Education in States and Nations" in Measuring What Students Learn, OECD, 1995

    "Education Finance Indicators: What Can We Learn From Comparing States With Nations?" in Developments in School Finance, NCES, January 1995.

    An Overview of the Korean Vocational Education System, PES, July 1994

    "American Public Opinion on Trade, 1950-1990," Business Economics, (v.28 n.3), July 1993

    "A Welcome Rain Falls in West Africa," The Christian Science Monitor, June 23, and on Monitor Radio, June 26-28, 1987.


    Not on the Web (TESTING):

    Phelps, R.P. (2012). The effect of testing on student achievement, 1910–2010. International Journal of Testing, 12(1), pp.21-43.

    "Estimating the Cost of Systemwide Student Testing in the United States," Journal of Education Finance (JEF), Winter 2000

    "Extent and Character of Systemwide Testing in the U.S." Educational Assessment, V.4, N.2, 1997

    "The Fractured Marketplace for Standardized Testing," book review, Economics of Education Review, (v.13 n.4) December, 1994

    "The Economics of Standardized Testing," delivered at the annual meeting of the American Education Finance Association, 1994

    "Benefit-Cost Analyses of Testing Programs," delivered at the annual meeting of the American Education Finance Association, 1994

    "National Testing, Pro and Con," The Education Digest, November, 1993.


    Not on the Web (EDUCATION FINANCE):

    National Assessment of Vocational Education: Funding and Accountability, PES

    "Estimating the Cost of Systemwide Student Testing in the United States," Journal of Education Finance (JEF), Winter 2000

    The Effect of University Host Community Size on State Growth," Economics of Education Review, v.17,n.2, 1998.

    Education System Benefits of U.S. Metric Conversion," Evaluation Review, Feb. 1996.

    "Education Finance Indicators: What Can We Learn From Comparing States With Nations?" in Developments in School Finance, NCES, January 1995.

    "The Fractured Marketplace for Standardized Testing," book review, Economics of Education Review, (v.13 n.4) December, 1994

    "The Economics of Standardized Testing," delivered at the annual meeting of the American Education Finance Association, 1994

    "Benefit-Cost Analyses of Testing Programs," delivered at the annual meeting of the American Education Finance Association, 1994

    "Education Finance in States and Nations," delivered at the annual meeting of the American Education Finance Association, 1994

    "Sensitivity Analysis of International Education Finance Indicators," delivered at the annual meeting of the American Education Finance Association, 1994


    Not on the Web (STATISTICAL/METHODOLOGY/EVALUATION):

    Longitudinal Data Analysis -- A Guide for NCES Data Users, Chapter 3, NCES

    Doctoral Scientists and Engineers in the United States, 1997, NSF, November 2000

    The Effect of University Host Community Size on State Growth," Economics of Education Review, v.17,n.2, 1998.

    Review of Survey Items on Continuing Education and Training, NCES-OECD, Feb. 1998

    Review of Proposed Indicators for Network B-Student Outcomes, NCES-OECD, July 1997

    "Student Achievement and Adult Literacy," chapter 6, Education at a Glance: OECD Indicators, Centre for Educational Research and Innovation, OECD, 1997

    "International Comparisons of Public Expenditures on Education," in: The Condition of Education, 1998, _1997, _1996, _1995, _1994, NCES

    Education System Benefits of U.S. Metric Conversion," Evaluation Review, Feb. 1996.