OpenAssessment.org – Our EdStartup Idea

The Idea
We* want to create a collaboratively developed database of openly licensed assessment items and measurement training. All of the items in the database will be written by trained educators and other subject-matter experts from various content areas and at various educational levels. The items will be aligned to educational standards (where applicable) and made available to educators in multiple formats, including PDF download for free, use within an internal delivery and analytics system, and use within external delivery systems. The openly licensed measurement training will also be freely available to all users.

The Problem
Teachers lack the time, resources, and expertise needed to create high quality classroom assessments. And even in the rare cases where time, resources, and expertise are not limited, typical classroom assessments still lack the number of responses necessary for the kinds of measurement analyses that could help teachers improve their classroom assessments in a systematic way.

The Solution
A collaboratively developed database of openly licensed assessment items will allow teachers to improve the quality of their classroom assessments by (a) providing them with increased access to higher quality test items, and (b) providing them with access to measurement training and support. All users will have free basic access to all items in the database because the items will be openly licensed. In addition, the collaborative approach to item development will decrease the individual time needed to create high quality assessments by capitalizing on the inputs of a community of subject-matter experts. Also, item writing training and psychometric support will increase teachers’ practical measurement skills. Finally, items used within the internal delivery and analytics system will have large enough response numbers to allow for more sophisticated psychometric analyses and item improvement.

The Why
High quality assessment of student learning is important for making sound pedagogical and administrative decisions. However, many important educational decisions are based on assessment results that aren’t highly valid or reliable. This is especially true at the classroom level. But this is not because many teachers don’t want to improve their assessments. Most teachers couldn’t do much about this problem even if they wanted to. They not only lack the funds and training, but they simply don’t have enough item responses to conduct many psychometric analyses. Indeed, many teachers lack the time, software, training, and secure server space to even maintain a database of items and their psychometric properties. By providing teachers with increased access to this kind of database and to practical measurement training we can improve the quality of assessment in our educational institutions.

*Current intellectual contributors to this idea include myself, David Wiley, T. Jared Robinson and Dan Allen.

6 thoughts on “OpenAssessment.org – Our EdStartup Idea

  1. I love the idea of free assessment for every teacher that meets the needs of the students. A few questions that I have (if you don’t mind).

    1. How do you set yourself apart from other assessment generating sites?
    2. Do you have your own team of educators that are generating these assessments?
    3. How qualified of educators do you have to be to be apart of this? (Because it sounds lucrative ((and I might want to be apart of it)).

    I agree with what you have and think it is a wonderful idea! A majority of teachers have a hard time struggling with lessons and not getting the full potential from their assessments. Good idea!

  2. Great questions, Trever.

    There are two main differences between OpenAssessment.org and other assessment generating sites. First, all items in our database will be openly licensed with a Creative Commons share-alike license (creativecommons.org). This means that all items can legally be reused, revised, remixed and redistributed. Second, OpenAssessment.org will be designed as a collaborative space where educators and other subject-matter experts can come together in a community to create and share items. Initially, we may offer some monetary incentive for item writing, but our long-term business model includes other incentive strategies, like offering access to item-level data and our test delivery system in exchange for a certain number of items written or for a fee (a freemium-type model).

    Ideally, our hope is that educators will see the inherent value in having acces to these higher quality items and the free measurement training and will “pay it forward” by contributing some items of their own.

    Qualifications for item writing are simply expertise in some subject area. No other qualifications needed.

  3. Assessment is the lever that will move the world. You get what you measure. And so on…

    I see the potential for two beautiful outcomes from this work:
    1. Meaningful assessment that is less reductionist and flawed than what we currently have in place
    2. Services for self-assessment built on top of this platform that would enable students to privately figure out “do I get it yet?”, at least when it comes to standardized subjects, and which would support self-paced and non-traditional learning. “If I can pass the test, I don’t have to submit to seat-time or other irrelevant requirements.”

    I also see risk:
    1. An increased focus and time spent on assessment to the exclusion of valuing those things that we don’t know how to measure (joy in learning, judgment, creativity, independence, etc.)
    2. Yet another tool for inauthentic accountability measures
    3. More reliance on extrinsic motivators that reduce students’ intrinsic motivations to learn

    1. Great points, Marie. Thank you for your feedback! I’m especially interested in your idea for a student-directed assessment tool on top of the database.

      In regards to your comments about risk:
      1. The goal of this database is not to increase teachers’ focus on assessment, but, rather, to help them improve the assessment they already do in the classroom.
      2. The discussion around the authenticity of the accountability measures in our schools (and in the educational measurement world more broadly) is a contentious one indeed. I would be happy to hear what you believe is an “authentic” accountability measure. However, we feel that our product would actually help teachers create more authentic assessments (we’re not just talking multiple choice items here) that are aligned with standards, yes, but also with what teachers feel is important personally for their students.
      3. We aren’t claiming to reduce the reliance on extrinsic motivators, but we are certainly not promoting more reliance. Our goal is simply to help teachers improve the assessments they already give and plan to give. If extrinsic motivators themselves are bad, then poor quality extrinsic motivators are even worse.

      1. “If extrinsic motivators themselves are bad, then poor quality extrinsic motivators are even worse.”

        A higher quality bad thing can be worse than a poor quality bad thing if it does the bad thing better. It may simplify the process of doing the bad thing until the bad thing is everywhere. One word: Instagram.

  4. Thanks for the comment, Steve. If I’m reading you correctly, you’re tapping into the age-old argument of whether testing and assessment themselves are good things. For transparency sake, this is what I believe about testing in education: assessment is vital for teachers to understand and adapt to their students’ learning. Tests are one very important way that students “share back” with teachers (see David Wiley’s argument that education=sharing). That said, it’s critical to understand that tests themselves can be high or low quality (or somewhere in between) in terms of psychometric properties. The goal is to create good tests (psychometrically speaking). However, even the best test can be used inappropriately. I am not a big fan of the high-stakes, large-scale tests given in our schools these days. This is not because these tests are low quality, but because people too often use the results inappropriately. The idea we are espousing in this post relates to classroom assessment, where it is hoped that test results will be used more appropriately to help students improve their learning and help teachers adjust their instruction (though this is not always the case). At the classroom level, psychometric quality of test items is an important issue (to be improved by our database). So is how teachers interpret and use test results (which will be covered in-depth in our open course). In short, no, I don’t think improving the quality of classroom tests is doing a bad thing better. I honestly believe it is doing a good thing better.

Leave a Reply

Your email address will not be published. Required fields are marked *