Illustrating the Pain

This simple figure illustrates the barrier between teachers and (a) assessment training and expertise, (b) access to higher quality test items, and (c) self-awareness about actual assessment expertise. The barrier is multi-faceted and includes availability, opportunity, time, and money. Our edstartup idea provides clear and innovative solutions to each of these facets.

Validating Our Assumptions

Today I chatted face-to-face with eight teachers from various schools and subject areas (as well as with a couple of people within a school of education) to verify that the assumptions we have made about teachers’ pain are accurate.

Through my interviews, I was able to confirm each of following seven basic assumptions:

1. Teachers lack assessment training.

2. Teacher training programs do not have curricular space (or priority) for assessment training.

3. Assessment development is time consuming and often tedious.

4. Teachers rely on test banks and other sources of assessment items that have unknown quality and alignment.

5. Many teachers aren’t doing much to solve this problem. However, some teachers have started collaborative assessment development activities at the department, school or district levels. This means that some teachers, at least, are looking for ways to increase the test development efficiency and quality.

6. Very few teachers obtain assessment training through professional development activities. Generally, teachers who do obtain some training are those who have been asked to help in the development of large-scale standardized tests. This kind of training has limited applicability at the classroom level.

7. Teachers have limited access to high quality test items, but many teachers desire more access to such items.

In addition, my conversations with these teachers (and other education experts) revealed at least one other critical reality: teachers often don’t know what they don’t know about assessment.

It turns out that most teachers feel that they have sufficient capability in assessment development and evaluation. However, a few teachers that I talked to who had received minimal training (and one currently enrolled in a measurement Ph.D. program who has received significant training) were emphatic that their assessment knowledge prior to training was exceedingly limited but that they weren’t aware of the fact. What this means is that it will be critical to find the best way to help teachers realize the benefit that could be gained by using our product. One teacher (who is training to become a principal) recommended focusing our marketing toward administrators who, if bought-in, could persuade their teachers to use it. A second way to increase use would be to offer some sort of credit (college or licensure) for those who complete the training.

Finally, most of the teachers I talked with gave very positive feedback about the idea itself, including saying things like “I would use your database” and “Your training and database would be so helpful to me and other teachers I know.”

Photo Credit: http://www.innovationtools.com

OpenAssessment.org Pain Test

The purpose of this post is to conduct the first part of a pain test on our edstartup idea. The general goal of a pain test is to better understand the specific pain that potential end-users of our idea are experiencing and thus refine our idea to more directly alleviate this pain. In this post, we answer six key questions that get at the underlying assumptions we have about the problem we are trying to solve.

1. What causes the problem?

There are several causes to the problem we are trying to solve.
First, teachers generally lack training and expertise in the assessment theory and practice. Teachers do not usually receive this training because assessment and measurement are often not the highest priorities for teacher preparation programs that focus more on pedagogy, curriculum development, and classroom management. Simply put, there just isn’t enough curricular room in pre-service training programs for courses that cover assessment and measurement in any useful depth or breadth. Teachers in higher education, on the other hand, rarely receive even a basic training in the teaching profession – let alone training in assessment.
Second, assessment development is time-consuming. Teachers who desire the highest quality assessments for their students will attempt to write tests themselves in order to align their assessments as much as possible with the curriculum in their classroom. The time spent writing assessments can be an incredible time sink. What’s more, without measurement expertise, these tests are often of suspect quality in terms of validity and reliability.

Third, assessment adoption is either very expensive or results in tests that are of unknown quality. Perhaps most often, teachers will adopt test items from other sources – like textbooks and test banks. Test banks containing high quality items are expensive (ETS spends upwards of $200 per item to develop their highest quality assessments – like the SAT and GRE – for example). In reality, most test banks are written by people with very little measurement expertise, and they are always written to broad, general standards (if at all). Rarely are items aligned specifically to any one set of state (let alone district or school or teacher) standards. As an example, one of our colleagues recalls with fondness his days as a graduate student when his advisor would pay his students $1 per item to help compile a test bank for a new textbook he had written. One can only guess how much attention was given to each item by those starving graduate students.

2. Think about the people with the problem. What are they currently doing, or willing to do, to solve it?

Teachers with this problem are generally not doing much to solve it, even if they want to. For starters, there are very few options for in-service or pre-service teachers to obtain training in measurement and assessment. Thus, we suspect that most teachers do not see this is a practically solvable problem or simply view low-quality test banks as one kind of solution. Despite this, some forward-thinking teachers have begun collaborating with colleagues within their school, district or institution on assessment development or item adoption initiatives.

3. What are all of the current solutions to the problem?

Current solutions to the problem of low assessment expertise among teachers are generally quite expensive. There are some assessment training workshops that teachers (or schools) can pay to attend, but these have deep costs in terms of fees, travel, and time.  We don’t know of any current solutions to the problem of inaccessibility to higher quality test items. The higher the quality, the more expensive the item. In all cases we know of, these costs are always passed on to the item user.

4. Why aren’t the current solutions good enough?

The expense of the current solutions is prohibitive, especially in the current economic climate. In fact, as the demand for high quality classroom assessment goes up, the money available to spend on training and materials for the creation of such assessment is going down.

5. How long has it been a problem?

Classroom teachers have always lacked training in assessment and measurement. However, the demand for high quality assessment is much more recent. Ever since the first education reform policies hit the ground in the 1960s (in the U.S. at least), the focus on measurable progress and assessment has increased with each subsequent policy revision. In addition, access to high quality test items has been prohibitively expensive for as long as such items have been available.

6. How easily could something change to make the problem go away?

This problem could be fairly easily solved with the use of open educational resources, a little bit of money, and current technologies – especially the collaborative potential of the Internet. You can read more about our proposed solution to this problem on a previous post.

With our general assumptions now explicated, our next step in the pain test is to conduct a little market research. Talking directly with teachers (and faculty in education programs) will help us verify these assumptions and make revisions to our ideas. Most importantly, this field test will help us know if the pain we’re trying to alleviate is as real and sharp as we assume.

OpenAssessment.org – Our EdStartup Idea

The Idea
We* want to create a collaboratively developed database of openly licensed assessment items and measurement training. All of the items in the database will be written by trained educators and other subject-matter experts from various content areas and at various educational levels. The items will be aligned to educational standards (where applicable) and made available to educators in multiple formats, including PDF download for free, use within an internal delivery and analytics system, and use within external delivery systems. The openly licensed measurement training will also be freely available to all users.

The Problem
Teachers lack the time, resources, and expertise needed to create high quality classroom assessments. And even in the rare cases where time, resources, and expertise are not limited, typical classroom assessments still lack the number of responses necessary for the kinds of measurement analyses that could help teachers improve their classroom assessments in a systematic way.

The Solution
A collaboratively developed database of openly licensed assessment items will allow teachers to improve the quality of their classroom assessments by (a) providing them with increased access to higher quality test items, and (b) providing them with access to measurement training and support. All users will have free basic access to all items in the database because the items will be openly licensed. In addition, the collaborative approach to item development will decrease the individual time needed to create high quality assessments by capitalizing on the inputs of a community of subject-matter experts. Also, item writing training and psychometric support will increase teachers’ practical measurement skills. Finally, items used within the internal delivery and analytics system will have large enough response numbers to allow for more sophisticated psychometric analyses and item improvement.

The Why
High quality assessment of student learning is important for making sound pedagogical and administrative decisions. However, many important educational decisions are based on assessment results that aren’t highly valid or reliable. This is especially true at the classroom level. But this is not because many teachers don’t want to improve their assessments. Most teachers couldn’t do much about this problem even if they wanted to. They not only lack the funds and training, but they simply don’t have enough item responses to conduct many psychometric analyses. Indeed, many teachers lack the time, software, training, and secure server space to even maintain a database of items and their psychometric properties. By providing teachers with increased access to this kind of database and to practical measurement training we can improve the quality of assessment in our educational institutions.

*Current intellectual contributors to this idea include myself, David Wiley, T. Jared Robinson and Dan Allen.

The EdTech Startup Space

The introductory material on EdStartup 101 really provided me with my first foray into the fast-paced and exciting world of educational technology startups. The amazing ingenuity and willingness to take risks shown by many of the founders of these companies is at once impressive and intimidating. I think to myself, how could I ever do that?! But, as I think more about my own edstartup idea (to be revealed in a later post) and how excited I am about getting it off the ground and into a space where it can be useful to educators and students, I think: You know what? I can be amazing too. I can handle the risks and the challenges. I’m passionate. I’m not afraid to fail because I know l can learn from failure. (Ok. I’m a little afraid to fail, but I hope to surround myself with others who aren’t).

I think about open education, assessment, and personalized learning, so some of the startups that are interesting to me include Coursera, Study Island, i-Ready, Knewton, and Naiku. These companies are fascinating in how they leverage the advantages of technology and theory to improve learning. Because I have a background in educational measurement, I’m especially interested in companies that apply sound psychometric theory to their learning analytics approach. It seems that Knewton (given their start as a test-prep company) does this quite well. I’d like to draw on their experience as I begin my own adventure in the world of ed tech startups.