OpenAssessment.org Pain Test

The purpose of this post is to conduct the first part of a pain test on our edstartup idea. The general goal of a pain test is to better understand the specific pain that potential end-users of our idea are experiencing and thus refine our idea to more directly alleviate this pain. In this post, we answer six key questions that get at the underlying assumptions we have about the problem we are trying to solve.

1. What causes the problem?

There are several causes to the problem we are trying to solve.
First, teachers generally lack training and expertise in the assessment theory and practice. Teachers do not usually receive this training because assessment and measurement are often not the highest priorities for teacher preparation programs that focus more on pedagogy, curriculum development, and classroom management. Simply put, there just isn’t enough curricular room in pre-service training programs for courses that cover assessment and measurement in any useful depth or breadth. Teachers in higher education, on the other hand, rarely receive even a basic training in the teaching profession – let alone training in assessment.
Second, assessment development is time-consuming. Teachers who desire the highest quality assessments for their students will attempt to write tests themselves in order to align their assessments as much as possible with the curriculum in their classroom. The time spent writing assessments can be an incredible time sink. What’s more, without measurement expertise, these tests are often of suspect quality in terms of validity and reliability.

Third, assessment adoption is either very expensive or results in tests that are of unknown quality. Perhaps most often, teachers will adopt test items from other sources – like textbooks and test banks. Test banks containing high quality items are expensive (ETS spends upwards of $200 per item to develop their highest quality assessments – like the SAT and GRE – for example). In reality, most test banks are written by people with very little measurement expertise, and they are always written to broad, general standards (if at all). Rarely are items aligned specifically to any one set of state (let alone district or school or teacher) standards. As an example, one of our colleagues recalls with fondness his days as a graduate student when his advisor would pay his students $1 per item to help compile a test bank for a new textbook he had written. One can only guess how much attention was given to each item by those starving graduate students.

2. Think about the people with the problem. What are they currently doing, or willing to do, to solve it?

Teachers with this problem are generally not doing much to solve it, even if they want to. For starters, there are very few options for in-service or pre-service teachers to obtain training in measurement and assessment. Thus, we suspect that most teachers do not see this is a practically solvable problem or simply view low-quality test banks as one kind of solution. Despite this, some forward-thinking teachers have begun collaborating with colleagues within their school, district or institution on assessment development or item adoption initiatives.

3. What are all of the current solutions to the problem?

Current solutions to the problem of low assessment expertise among teachers are generally quite expensive. There are some assessment training workshops that teachers (or schools) can pay to attend, but these have deep costs in terms of fees, travel, and time.  We don’t know of any current solutions to the problem of inaccessibility to higher quality test items. The higher the quality, the more expensive the item. In all cases we know of, these costs are always passed on to the item user.

4. Why aren’t the current solutions good enough?

The expense of the current solutions is prohibitive, especially in the current economic climate. In fact, as the demand for high quality classroom assessment goes up, the money available to spend on training and materials for the creation of such assessment is going down.

5. How long has it been a problem?

Classroom teachers have always lacked training in assessment and measurement. However, the demand for high quality assessment is much more recent. Ever since the first education reform policies hit the ground in the 1960s (in the U.S. at least), the focus on measurable progress and assessment has increased with each subsequent policy revision. In addition, access to high quality test items has been prohibitively expensive for as long as such items have been available.

6. How easily could something change to make the problem go away?

This problem could be fairly easily solved with the use of open educational resources, a little bit of money, and current technologies – especially the collaborative potential of the Internet. You can read more about our proposed solution to this problem on a previous post.

With our general assumptions now explicated, our next step in the pain test is to conduct a little market research. Talking directly with teachers (and faculty in education programs) will help us verify these assumptions and make revisions to our ideas. Most importantly, this field test will help us know if the pain we’re trying to alleviate is as real and sharp as we assume.

1 thought on “OpenAssessment.org Pain Test

  1. Other solutions to the problem include simply using poor assessments, using the assessments provided by curriculum guides or textbooks, substituting dichotomous items for open-response items, and purchasing items always is an option.

Leave a Reply

Your email address will not be published. Required fields are marked *