Open Ed 2012 : Preliminary Results of Testing the OERPUB Editor designs

Chinese food in Vancouver

Open Ed 2012 just closed in Vancouver and I wrote this on my way home while looking forward to the community of family. I am extraordinarily lucky to work with a group of smart, passionate, and creative colleagues, whose hard work paid off for us handsomely in Vancouver. We were bringing designs for what we hope is a sustainable approach to authoring education resources that can be remixed, adapted, improved, and extended.

Since Open Ed attracts a diverse community of educators, technologists, and visionaries, we brought with us two things we wanted to vet. We wanted to learn from this group and attract partners to our effort, and we have succeeded in both. First, we brought high-fidelity (which here means it looks great and parts of it work :-)) mockups to test with authors at the conference. Secondly, we brought a technical architecture and working prototype for delivering the features in the mockup as extensible, open-source software. In addition to me, two of our user interface designers, Max Starkenburg and Adrian Garcia, came and also acted as test monitors and test analysts. We recruited Megan Beckett from Siyavula to our cause since she was already coming to talk about Siyavula's textbooks, community authoring, and badges. She helped beta-test and refine the tests, and then even acted as a test monitor for one of the days. To convey the technical architecture, two of the leading developers on the project, Marvin Reimer and Phil Schatz, joined and talked with anyone who cared about the technology, which turned out to be a substantial number of people and projects at the conference. 

Testing the interface at OpenEd.

We had close to 30 people go through the tests of the editor! People spent between 15 and 45 minutes helping test. For the curious, here are the tasks they were given.

What we learned. Well, Adrian and Max and Megan have lots of data we haven't yet gone through, yet, but we spent Thursday evening after the conference working through the highlights. Our overall System Usability Score was a solid B, which seems like exactly where we should be right now. Our ambitions are sky high and the best design principals have to be tested.  

Analyzing the results (qualitative)

Things that tested well : the approach to math editing, the math cheat sheet, the drag and dropping of semantic features, the highlighting box and options gear-icon.

Things that need some rethinking (luckily with a whole lot of user data to inform them): the exact way we presented image sourcing information was confusing and needs a rework. Some people loved it, but some people didn't understand it. One of the semantic features we tested was "exercises", the Connexions inspired structured question/answer format. The naming of the parts wasn't intuitive. The other interesting problem was that in the editor the solution to the question shows directly below the question. Authors didn't like that one bit, because they don't want readers to see the solution right there. Earlier designs hid the solution, but they were jumpy, so we need to figure out how to cleanly show that the solution will be hidden from readers. We had a major problem with table editing -- no one could find how to add/delete rows. This one is so easy to fix, though, because we know everywhere people looked for it and we will just make those work. That is the beauty of this kind of testing.

Engaging new partners: Since we have designed the editor so that it can be embedded in a wide-variety of projects, built it on Aloha, an HTML5 editor with active contributors, and since the underlying format of the content is HTML5, we were successful in enticing the serious interest of other projects at the conference. There is still a lot of hard work to successfully engage participation and move forward. I am looking forward to that process. It has been my dream for this project since the beginning.