Marking student work eletronically, where is the product we want?

We have been using Turnitin for students to submit work since 2004. However, they still submit a paper copy and we still use a pen to mark it. Increasingly this is becoming frustrating futile. Students complain about the printing costs we’ve shifted to them, they have to find places to hand in work and get a receipt during office hours despite having handed in an electronic copy of their work at their convenience. Staff spend wasted hours providing useful feedback that we are never sure that students read (see the large piles of uncollected work we have sitting around). There must be a better way.

Given my long experience with Turnitin, I’ve dabbled in the previous version of Grademark which is the online marking system bundled with the plagiarism detection system. We dismissed it previously because it provided no bulk download facility for us to keep copies of marked work and the version that students could download offline was far inferior to the online version.

There have been lots of discussions floating around the university about electronic marking, so Jon Scott and I tried hard to think about using the system and spent an hour having a good play with it. Jon and I both mark electronically for the Open University as associate lecturers so we know the advantages and are happy to mark on-screen. However, after our playtime, we still have some major stumbling blocks to surmount with Grademark.

1. Lack of alignment of feedback and grades. One the benefits of grademark is that marks and feedback are released to students on a particular date, negating the need to return work manually to students. The major downside is that the grade is the first thing that they see, before they get to the feedback, if we use Grademark it will also push grades through the grade centre in Blackboard and students can see these in ‘my grades’. I know Alan Cann (@ajcann)agrees that this is misalignment of feedback and grade and is not helpful and does not encourage feedback to be read by the widest possible audience.

2. Difficulty in finding the feedback. If students access the feedback through their Turnitin Inbox in Blackboard, the route is even more tortuous. More clicks to get at your feedback than you can shake a stick at…(shown in video – no sound)

3. Integration with Dragon Dictate. Jon uses dragon dictate to insert coloured text into student essays when marking online. Pretty neat and it saves him time and RSI. We did manage to add text to one comment using dragon but then it failed after that. Not all staff will want to mark this way, but it would be nice if those that did could have the option.

4. Offline marking. I know that this is under development, but without it many staff will feel too tied to an online environment for widespread adoption.

I wondered if other users of grademark have already sorted these issues – please let me know if you have!

Other options for electronic marking:

As we use Blackboard (v 9.0) another option may be for us to use the Sheffield Hallam assignment handler [links to PDF]. This seems to offer what we want, and seems very similar in functionality to the Open University Assignment handling system. Staff can bulk download the submitted work from students, annotate an excel spreadsheet with feedback and grades and bulk return annotated files back to students. There is a very useful knowledge base on using the system available publicly. It looks like it has recently been updated too. This is a feature that is available via Blackboard if we want it, perhaps it would be worth investigating?

There seem to be lots of solutions out there for electronic marking but most seem to fall short of the relatively simple tools we need.

Comparison of PeerMark and Blackboard Peer Assessment tool

Following my presentation yesterday I wanted to expland on my experiences of testing the two peer marking tools available to us at Leicester University (I will report on my use of google presenter + chat used in the presentation in another blog post).

Technical notes: we are using Blackboard 9.0 with Turnitin fully integrated in to the VLE. The Peer and Self Assessment tool is the same as was included in Bb 8.0, following development by Dundee University.

1. Set up

Blackboard Peer and Self Assessment Tool: Fairly straight forward to set up, with pretty good notes explaining what all the terms mean. If you think in terms of the student work being submitted as short answer questions, each of which can be peer assessed under a set of criteria, then set-up is straight forward.

PeerMark: I found setting this up very confusing, the interface has very little support or instruction and it is not at all clear what is set where. I always seem to get an error asking for latest submission dates to be set when I haven’t chosen late submissions to be allowed. Perhaps this is related to the VLE integration, but it is very confusing. There are several places where you need to click on ‘more details’ or a a further link which enables you to add what I thought was essential information rather than ‘optional’ extra information. See video.

2. Student use

Testing these types of assignments is pretty hard to do on your own, you need at least two peers to help you. Unless work has been submitted to assignment you can’t see the peer review element, and unless peer reviews have been submitted you can’t see how the results would look. Blackboard wins here as it has a ‘preview’ feature which does help you to geel confident about how instructions will appear to the students.

Several colleagues commented that the Blackboard submission and review process was very ‘click’ heavy (which it is) and very difficult to know what to do when. When accessing the evaulation section it is not at all obvious how to open the work that needs to be peer assessed and several colleagues didn’t realise they had work to review.

In comparison, PeerMark is well thought out from teh student user point of view. Having the peer work and criteria on one page makes the process simple to complete. A status bar at the top of the page shows when the review is 100% complete and can be submitted.

One colleague commented that finding the feedback from the peer reviews was very difficult in PeerMark, buried under several layers.

3. Instructor evaluation of the results

Blackboard wins out slightly here, as marks given by peers were easily seen on a results list, and submissions and evaluations were easy to access. PeerMark was more cumbersome and not as easy to see which students had submitted work and what their comments were.

4. Features and application

I had compare the two services in my presentation:

comparison of features of PeerMark and Blackboard peer assessment

PeerMark was much more flexible in terms of the ways that peers could be allocated to review other student work and in a striaght comparison would win hands-down if it weren’t for the awful set-up interface and confusing results section (for staff and students). As it is, it’s probably a tie and either tool could be used depending on the task at hand.

5. Videos

I made a series of videos to demonstrate staff and student views of PeerMark and the Blackboard tool, essentially because I couldn’t trust the tech in my presentation, but also for using with instructors as neither tool is particularly easy to explain!

PeerMark

Setting up PeerMark (2:23)

Student reviewing work in PeerMark (2:28)

Instructor view of PeerMark results and evaluations (0:45)

Blackboard Peer and Self Assessment

Setting up a Peer Assessment in Blackboard (3:50)

Student submission of work to Blackboard peer assessment tool (0:53)

Peer review by student of Blackboard assignment (1:38)

Instructor view of Blackboard peer review tool results, evaluations and student submissions (1:54)

Help with Blackboard 9

Blackboard 9 course

Although we are still awaiting an official announcement, it is widely accepted that we will be upgrading from Blackboard 7.3 to Blackboard 9.0 this summer (geek note: we are Blackboard ‘original’ flavour, not ex-WebCT). As the front line support for our staff using Blackboard in the School of Biological Sciences I am of course spending time looking at the test server installation. I need to see how our courses look in the new format and try to work out what support our staff might need. I’m going to blog about any problems I have for two reasons: firstly, I can’t remember anything these days unless I write it down and secondly, I live in hope that someone will point out the error of my ways and propose a solution to my problems.

It occurs to me that if I only blog the problems the outside impression will be that I am moaning about Blackboard 9, I don’t really want to come across that way, but I think it is the quickest way for me to get help and I hope that my readers will bear in mind that I am just trying to crowdsource the best help I can to provide the best service I can to our staff. So, with that in mind:

1. List all Users and finding all students (solved)

I suspect I will spend a lot of time asking where the button or menu item for such and such has gone from the version I’ve grown to love and hate over the last 5 years. here is the first one:

We do manual enrolments (or batch upload but there is still a live person involved). It is therefore really helpful to be able to quickly list the number of students enrolled on the course. This used to be simple, go to the gradebook and look at the number of the bottom of the page (e.g. ‘104 users’ displayed). The gradecentre is very swish and userfriendly, however there is no equivalent total listing of students. I can list ALL users (in a weird workaround where I have to list all users ’email’ ‘not blank’) but this list isn’t sortable by role. Equally this means I can’t quickly see how many staff are on the course, which I used to be able to do with a building block for ‘list all users’.

Update 14.05.10 : Alan found the solution:

Tools (not listed under course tools on the control panel, but under the tools on the menu item) lets you list all the ‘users’ which equates to students. This has a total number at the bottom.

Tools:Register

One problem solved, onto the next!

anti? social #durbbu10

Despite the snow, the team at Durham put on another great Blackboard Users conference this year. My main reason for attending was to meet some of the brave souls who made the jump to Blackboard 9.0 in the summer. Leicester was considering a move from 7.3 to 9.0 but pulled out in August 2009. I work during school term times only, so last June and July I was busy blogging in preparation for the move. I don’t think much about who reads the School of Biological Sciences blog, as I mainly use it as a tool for me to keep instructions that I send people and a list of FAQs that save me typing out the same responses to the same emails every week. I was therefore rather surprised when Natalie Lafferty told me that she found the site really useful to gain information about Blackboard 9 in preparation for University of Dundee’s impending upgrade.

The panel discussion session was one of the most interesting sessions during the conference.

The cat that had 9 lives: Blackboard Release 9 upgrade experiences
Jake Gannon, University of Liverpool
Hannah Whaley, University of Dundee
Rachel Fitzgerald, University of Northampton
Russell Smeaton, University of Teesside
Malcolm Murray, University of Durham

The five universities represented had certainly had a pretty tough time with 9.0. The main messages were:

  1. All had a well laid out procedure for making the decision to upgrade. Most had either asked their users (staff and students) directly about Blackboard 9 or used feedback from annual surveys to inform their decision. Northampton carried out user testing with their staff and students on 9.0 and the response was very positive.
  2. All had periods of downtime in the first weeks of the academic year (2009/10) which caused problems severe enough to have senior management seriously question the decision to upgrade. This has prompted strategic reviews of eLearning in several institutions and much soul searching about why we use VLEs.
  3. All now have a stable product, though with some minor bugs still outstanding.
  4. Everyone agreed that the now stable system affords a much better user experience, especially in the area of group work tools but that the damage done by serious performance issues at the start of term had overwhelmed the potential of new features to impress.
  5. Browser issues provide serious cause for concern. Now running in Java, one of the difficulties faced was a lack of java memory once the system was hit with a significant number of users at the beginning of term. Test servers did not replicate the real-world loads placed on the system. The main advice here was to make sure that the hardware was up to the job and to get the java settings checked by as many people in Blackboard as possible before launch.
  6. Another browser problem is the requirement for IE 7. In particular, Dundee med school had problems with their staff working in the NHS who are limited to using IE 6 only (this is common across the NHS as far as I know). Without IE 7 staff cannot use the drop down menus that drive most of the editing facilities in Bb 9.
  7. Final browser problem was the intolerance of the system to old browsers and slow internet connections. Rather than just take ages to load, the system dumps the connection and will not load at all. Could be a real problem for distance learners on dial up or ropey connections.
  8. The most worrying point for me was that none of the Universities felt that they had blazed the trail for the rest of us to follow. It is common for us at Leicester to wait for the first service to be applied to a ‘.0’ release before upgrading. Those that had made the upgrade had all had individual patches or fixes made for them that were specific not only to their systems, but to the legacy of use of Bb they brought with them (particular combinations of adaptive release settings, recycling methods, previous upgrade paths etc). As far as I could understand this meant that a 9.0 SP1 would not be guaranteed to be a stable product.

Finally, it is worth noting that of all the Universities involved, Northampton ran on a service hosted by Blackboard. They seemed to have had less downtime than the other HEIs (though an unacceptable amount for a hosted system) and had a less negative experience overall. This seemed to be down to a large amount of preparation on their part, starting in December 2008 for a Sept 2009 launch with training, testing and information for staff and students very widely and thoroughly disseminated. Perhaps the fact that they were hosted meant that the blame for any downtime was deflected to Blackboard themselves, rather directed at the ITS team which helped them to maintain a good relationship with their users and encourage them to try the new 9 features.

Do you come here often? The fleeting nature of communication in a 140 character World #durbbu10

I am in the snowy frozen north for the next two days at the Blackboard UK users conference in Durham. I am presenting our work on encouraging our first year undergraduates to think about their personal learning environments and how this may have influenced their means of communication with us as instructors on the course.

The presentation blends two projects that involved Alex Moseley, Alan Cann and Stuart Johnson (all at University of Leicester) and myself.

Abstract: A first year undergraduate IT and numeracy key skills module on Blackboard (v 7.3) delivered to over 200 students over two semesters has made use of innovative online assessments over the last 10 years. The IT section of this module was substantially revised in 2008/9 to assist students with the concepts and competencies of information literacy, ultimately leading towards the construction of a personal learning environment (PLE) and a reflective e-portfolio (Badge et. al. 2009).  This was achieved by the introduction of freely available Web 2.0 tools. All the course content is delivered wholly online, including marking (EMCQs,  see Cann, 2005, Google Documents, delicious, Google Reader, see Badge et. al. 2009)  and feedback (via YouTube videos).  A Blackboard discussion board has supported this course as a place for students to ask questions about the content and any administrative details since 2002. For the first time in 2008/9 we introduced Twitter to the course and students were encouraged to use Twitter to ask for help. The discussion board was still available but questions posed here were markedly less than in previous years (~100 messages per year previously, this year, zero). A small cohort of students used Twitter to ask questions about the course, stimulated in part by our study on Twitter and the student experience (Cann et. al. 2009). Now in the second year of using Twitter to support this course, this has become an accepted channel for students to contact the convenor. The discussion board is checked regularly but has not been used at all by students this year. Despite this course requiring students to access Blackboard at a minimum of twice per week, students are still not using it as a communication channel. How does this plethora of parallel communication channels affect the way staff/students will interact with Blackboard in the future? How will adding Google Wave to the mix affect things? Where is Blackboard in the era of the realtime web?

References:

Cann, A., Badge, J., Johnson, S., and Moseley, A. (2009). Twittering the student experience. ALT-N, 17.

Cann, A. (2005). Extended matching sets questions for online numeracy assessments: a case study. Assessment and Evaluation in Higher Education, pages 633-640. Citeulike citation

Badge, J. L., Johnson, S., Scott, J. S., and Cann, A. J. (2009). Encouraging lifelong learning habits in a web 2.0 enabled PLE. In Higher Education Academy Annual Conference. CiteUlike citation

Aargh! Turnitin fail

I’ve had my head against a brick wall today with several technical problems. The main one being that in some of our blackboard courses, Turnitin is not functioning, you can’t view old assignments, modify them or create new assignments. It turns out that this is because we were at the bleeding edge of using turnitin and we’ve been expired!

It’s rather tortuous but can be fixed by Turnitin, so if this has happened to you, get your admin person to contact iParadigms.

Way back in the mists of time (well it was 2004), we at University of Leicester installed an early version of the plug-in that connected Blackboard and Turnitin (version 1.something, I think). When a Turnitin Assignment is created on a Blackboard course using this plug-in, a corresponding class is created in Turnitin. This class is the container for the individual assignments and is a related to the stand alone version of Turnitin which operates like a mini VLE. This class container is never seen by the Blackboard user (instructor nor student), as they operate at the assignment level. the Blackboard module is their ‘class’ equivalent.  The old plug-in set expiration dates on these classes, and having been around long enough, our Blackboard courses that first tried the Turnitin integrated system back in 2004/5 have now expired and so cannot use Turnitin in any shape or form. This is also a consequence of us recycling our Blackboard courses, keeping the content the same but refreshing the student cohort each year.

The good news is that this can be fixed by the Turnitin support staff in the States, who can renew the classes and set them to never expire, but it’s not something your institutional admin can sort out. the bad news is that until they fix it, these Blackbaord courses can’t use Turnitin and term starts on Monday. Fingers crossed for a quick response from the States!

Update: Monday 28 Sept. No it’s worse than that – response from TII is: ‘iParadigms have told us that all classes that are created have a maximum duration of 5 years. after this point they will expire and will not be able to be extended.’ This is a major problem for anyone that recycles their blackboard courses year on year and uses TII integration. All your courses will have a 5 year life span and then they will have to be re-made to reset the expiration date.  If this affects your institution please raise a support ticket to TII to ask them to rethink this policy. Emailing directly to requktii@turnitin.com. If this does not work, you can use the form at https://submit.ac.uk/help/helpdesk.asp