Magnitude of Task-Sampling Variability in Performance Assessment: A Meta-Analysis

Research output: Contribution to journalArticle

11 Citations (Scopus)

Abstract

This study examined the percentage of task-sampling variability in performance assessment via a meta-analysis. In total, 50 studies containing 130 independent data sets were analyzed. Overall results indicate that the percentage of variance for (a) differential difficulty of task was roughly 12% and (b) examinee's differential performance of the same task was approximately 26%. Based on moderator analysis, research design and subject area were significant predictors of variance component for tasks (σ2 t) and variance component for person-task interaction (σ2 pt). Mathematics and listening were associated with relatively high proportion of task-sampling variability whereas foreign language was least affected by this variability. To reduce task variation, some techniques are recommended: (a) incorporating more facets when possible, (b) using a crossed rather than a nested design, and (c) incorporating occasion as a facet.

Original languageEnglish
Pages (from-to)887-912
Number of pages26
JournalEducational and Psychological Measurement
Volume69
Issue number6
DOIs
Publication statusPublished - 2009 Jan 1

Fingerprint

Research Subjects
Performance Assessment
Mathematics
performance assessment
Task Performance and Analysis
Meta-Analysis
Research Design
Language
Sampling
Facet
Moderators
Percentage
Nested Design
Components of Variance
Variance Components
moderator
foreign language
research planning
Predictors
Person

All Science Journal Classification (ASJC) codes

  • Education
  • Developmental and Educational Psychology
  • Applied Psychology
  • Applied Mathematics

Cite this

@article{aae016f6c20a4d4186e239808d38e8d8,
title = "Magnitude of Task-Sampling Variability in Performance Assessment: A Meta-Analysis",
abstract = "This study examined the percentage of task-sampling variability in performance assessment via a meta-analysis. In total, 50 studies containing 130 independent data sets were analyzed. Overall results indicate that the percentage of variance for (a) differential difficulty of task was roughly 12{\%} and (b) examinee's differential performance of the same task was approximately 26{\%}. Based on moderator analysis, research design and subject area were significant predictors of variance component for tasks (σ2 t) and variance component for person-task interaction (σ2 pt). Mathematics and listening were associated with relatively high proportion of task-sampling variability whereas foreign language was least affected by this variability. To reduce task variation, some techniques are recommended: (a) incorporating more facets when possible, (b) using a crossed rather than a nested design, and (c) incorporating occasion as a facet.",
author = "Chiung-jung Huang",
year = "2009",
month = "1",
day = "1",
doi = "10.1177/0013164409344550",
language = "English",
volume = "69",
pages = "887--912",
journal = "Educational and Psychological Measurement",
issn = "0013-1644",
publisher = "SAGE Publications Inc.",
number = "6",

}

Magnitude of Task-Sampling Variability in Performance Assessment : A Meta-Analysis. / Huang, Chiung-jung.

In: Educational and Psychological Measurement, Vol. 69, No. 6, 01.01.2009, p. 887-912.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Magnitude of Task-Sampling Variability in Performance Assessment

T2 - A Meta-Analysis

AU - Huang, Chiung-jung

PY - 2009/1/1

Y1 - 2009/1/1

N2 - This study examined the percentage of task-sampling variability in performance assessment via a meta-analysis. In total, 50 studies containing 130 independent data sets were analyzed. Overall results indicate that the percentage of variance for (a) differential difficulty of task was roughly 12% and (b) examinee's differential performance of the same task was approximately 26%. Based on moderator analysis, research design and subject area were significant predictors of variance component for tasks (σ2 t) and variance component for person-task interaction (σ2 pt). Mathematics and listening were associated with relatively high proportion of task-sampling variability whereas foreign language was least affected by this variability. To reduce task variation, some techniques are recommended: (a) incorporating more facets when possible, (b) using a crossed rather than a nested design, and (c) incorporating occasion as a facet.

AB - This study examined the percentage of task-sampling variability in performance assessment via a meta-analysis. In total, 50 studies containing 130 independent data sets were analyzed. Overall results indicate that the percentage of variance for (a) differential difficulty of task was roughly 12% and (b) examinee's differential performance of the same task was approximately 26%. Based on moderator analysis, research design and subject area were significant predictors of variance component for tasks (σ2 t) and variance component for person-task interaction (σ2 pt). Mathematics and listening were associated with relatively high proportion of task-sampling variability whereas foreign language was least affected by this variability. To reduce task variation, some techniques are recommended: (a) incorporating more facets when possible, (b) using a crossed rather than a nested design, and (c) incorporating occasion as a facet.

UR - http://www.scopus.com/inward/record.url?scp=70449969432&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=70449969432&partnerID=8YFLogxK

U2 - 10.1177/0013164409344550

DO - 10.1177/0013164409344550

M3 - Article

AN - SCOPUS:70449969432

VL - 69

SP - 887

EP - 912

JO - Educational and Psychological Measurement

JF - Educational and Psychological Measurement

SN - 0013-1644

IS - 6

ER -