Task 3 - Evaluation

The third task in the Introduction to TEL workshop was an evaluation of the group project (which is currently ongoing). I've got a few thoughts, which I'm keeping here as notes

What was good about it?
Uesful to see something come together, fairly rapidly (more so that firing emails back and forth, and a more organised way of working on a shared document (no concerns about working on an older version etc).

What was bad about it?
An early mistake on my part - researching formative rather than summative assessment. Probably caused by skim-reading the posts in the discussion thread, this probably wouldn't have happened face to face.
Communication feels a little stilted and formal. If something has been dropped (e.g. a question not answered), it feels forced to bring it up again.

What worked?

Somebody has to start the work, when somebody does, things can move quite well, but it seemed that especially where there is a little confusion or lack of clarity, that people can wait for somebody else to start. This happens face to face, but the asychronous communication probably makes it worse. You don't have any clues as to if a colleague is sat at their computer waiting for something, or if they've wandered off for lunch.

What did not work?
What would you do differently next time?

Did everyone pull their weight?
There did appear to be differences in the amount of effort or attention paid to the task, however this might be due to differing levels of familiarity with both the material and the enviornment. One of the facets of online communication often observed elsewhere is that people who are active participants, active in discussions, and who are 'early movers' can often exert substantial influence over the process.

Summative assessment

Summative assessment is what most obviously comes to mind in relation to 'assessment'. It is "the type of assessment that typically comes at the end of a module or section of learning and awards the learner with a final mark or grade for that section. The information about the learner is often used by third parties to inform decisions about the learner's abilities." (Fry et all, p.511)
Summative assessment tends to "define what students regard as important, how they spend their time and how they come to see themselves as students and then as graduates" (Brown et al 1997). It is therefore vital that summative assessment methods are aligned with the intented learning outcomes of the course.

Summative assessment can be both quantitative and qualitative. In contrast to formative assessment, the main roles of summative assessment in relation to the QAA code of practice are
Measurement, Standardisation and Certification. (QAA 2006:4) The QAA also provides six basic principles of assessment in terms of marking and grading. These should apply to any summative assessment use in virtual learning environments.

1. Consistency - marking and grading is appropriate and comparable across departments and faculties, through institutional guidance.
   
2. reliability - two or more markers would assign the same mark to a piece of work.
3. Validity - marking measures what it is supposed to measure.
   
4. Levelness - appropriate learning outcomes for level of qualification.
5. Transparency - aligned with perceptions of fairness, assessment criteria and marking schemes are open and available, tasks are published in good time, and there is a fair complaints procedure.
   
6. Inclusivity - able to make reasonable adjustments in assessing students with disabilities.
   

The following is from Biggs 2002
Assessment modes: (like to assess)

Prose type:
Essay exam (rote, question spoting, speed structuring)
Open book (as for exam, but less memory, coverage)
Assignment (read widely, interrelate, organise, apply, copy)

Quantitaitive/objective
Multiple choice test (recognition, strategy, comprehension, coverage)
ordered outcome (hierarchies of understanding)

Practical
Practicum (skills needed in real life)
seminar, presentation (communication skills)
posters (concentration on relevance, application, design)
interviewing (responding interactively)
critical incidents (reflection, application, sense of relevance)
project (application, research skills)
reflective journal (research, application, sense of relevance)
case study, problems (application, professional skills)
Portfolio (reflection, creativity, unintended outcomes)

Rapid
Concept maps (converage, relationships)
Venn diagramms (relationships)
Three minute essay (level of understnaidng, sense of relevance)
gobbets (realising the importance of significant detail)
short answer (recall units of information, coverage)
letter to a friend (holistic understanding, application, reflection).

Formative Assessment research

"one of the most important aspects of supporting student learning is the feedback that students received on their work. A not uncommon fault, particularly within a semester system is that students only find out how well, or how badly, they have done when their assessed work is rerutned with a comment and a mark at the end of the semester. By that time it is too late to take any remedial action...."

"There is no easy answer to this problem, but some suggested solutions may include the following. Students might submit a part of the final assessed work midway through the term, or they submit their planning work. Alternatively, a short piece of assessed work can be set for early in the semester with a return date before the final assesed work is completed. In some subjects online assessments can be used which can be marketd electronically to provide rapid feedback to students on their progress. Such assessments may be done in the students' own time and feedback is provided automatically. Peers and self-assessment can also be useful for providing feedback on learning if these are well structured and the assessment criteria are well understood - for example by discussing these with students." (Fry et al, 2009:120)

"assesment for learning places more emphasis on the formative" (135)
need for all methods to be relevant to ILOs

"formative assessment is defined as that which enables students to see how well they are progressing and gives them feedback. It perfeclty possible for summative assessment to have a formative component." (137)

So formative assessment is reliant upon good feedback.
which should (Yorke 2004) (the forth point is particularly applicable to formative)

• facilitate the development of self assessment (reflection) in learning
• promote peer and tutor dialogue around learning
• help clarify what good performance is (goals, criteria, expected standards)
• provide opportunities to close the gap between current and desired performance
• delivers high quality information to students about their learning
• encourages positive motivational beliefs and self-esteem
• provide information to teachers that may be used to help shape the teaching

formative and summative assessment also occur in the evaluation of teaching, as well as students. (198)

Self and peer assessment are increasingly seen as useful forms of formative assessment. They might also have some use within a VLE, given that making written work accessible to others (including in an early, pre-submission form) becomes feasible.

might have to balance this against plagarism risks.

Online environments might also be a good way to develop a portfolio of work (this could move across modules or indeed across whole courses).

Formative assessment might give the opportunity to use more innovative methods of assessment without the risk of it seeming unfair, or unpredictable to students.

Formative assessment need not be massive tasks, but can be quite brief - quizes, multiple choice, self-reflection.

Good practice in designing e-learning courses

1. Background - aims for the course
2. Intended learning outcomes - what do you want students to learn?
3. assessment - what assessment framework meets those aims (apparently, we tend to think about this too late, and students tend to think about if first - what do I need to do to pass?)
4. Content - what topics, what to read, in what order - this is presumably the 'traditional' bit we think of when 'writing a lecture'.
5. course structure -
6. Teaching and learning design - what are you going to do? what exercises/activities?
7. classroom - outcomes for particular sessions - this is probably going to be a bit different online, but there should probably be an equivalent step.
8. evaluation - how are we going to evaluate the quality or sucess of this course (and capture things we might need to change in the future)?

The Cranfield version is cyclical and includes

1. needs analysis
   
2. aims and outcomes
3. course structure
4. course content
5. Learning design
6. student and tutor support systems
7. assessment and procedures
8. development
9. implementation
10. evaluation
    

checking out some OU Courses

yeah, that's pretty neat.
doesn't mean much to me though as I'm not in any way an engineer (maybe I'll work through this and become one).

There's a lot of tools here too.

translating a face to face course online

The next task is to select a topic or key concept from a course you are currently teaching face-to-face and spend a a few minutes thinking about how this might translate into standalone interactive learning materials. Also to do this in relation to the following guidelines on good quality material:

• Help the learners find their way into and around your subject -- by by-passing or repeating sections where appropriate.
• Tell them what they need to be able to do before tackling the material.
• Make clear what they should be able to do on completion of the material -- e.g. in terms of objectives.
• Advise them on how to tackle the work -- e.g. how much time to allow for different sections, how to plan for an assignment, etc.
• Explain the subject matter in such a way that learners can relate it to what they know already.
• Encourage them sufficiently to make whatever effor is needed in coming to grips with the subject.
• Engage them in exercises and activities that cause them to work with the subject-matter -- rather than merely reading about it.
• Give the learners feedback on these exercises and activities -- enabling them to judge for themselves whether they are learning successfully.
• Help them to sum up and perhaps reflect on their learning at the end of the lesson.

(Rowntree 1990, pp. 82-83)


Some things come to mind around the topic of online privacy, which relates quite nicely to teaching in an online environment (although ideally one that is not so locked down that you can't access Facebook, Twitter or similar. You could run a number of interesting exercises that, for example including a google search by the student for their own online profile - to see how much information is available about them online - follower by discussion or possible (careful, edited if they like) group comparison.

There's plenty of material out there that can be straightforwardly read or acessed, so that's not a limitation.

My concern with this is frontloading - as I said previously, its very important that everybody knows what they are doing, but also you can easily loose an initital 'keenness' to participate if you have to read and remember a whole course's worth of instructions at the start. A good model to draw upon might be the early levels of many contemporary video games. Nobody learns how to play these by reading the manual, but rather the early levels are set up to teach you how to play the game - in line with a lot of education theory - they get you to perform small tasks that building up to more complex ones, there's lots of obvious feeback if actions work or not.

Structure's important - I've had to jump around a little bit to find the next task in this module today. And I think I'm spending more time on things than I might need to - that said, I'm getting something out of writing.

Communicating in a VLE

drawing upon information from JISC on computer mediated conferencing

God, there's a lot of material on here about how to communicate online. Actually, a lot of this feels familiar - I've spent time communicating online, and have bumped into a few of it's perils, and I've also discussed these with friends and colleagues.

I've learnt this elsewhere, but its incredibly applicable to education - given the absence of face to face interaction, and asychronous communication, it's vitally important that the shared norms of an educational space (how often communication should occur, between who, at what rate, in what volume, what level of critique is acceptable etc) should be really, very explicit and clear. In communication we take a lot of things for granted, which we probably can't do to the same extent in a VLE. It takes work to construct and maintain these norms of communication (and a hell of a lot of forum posts are actually serving this purpose). Also if people don't know what they're supposed to be doing, it very rapidly breaks down.

From a surveillance perspective, the permanence of the record is somewhat unsettling - particularly for education in politics (and possibly security). I'd probably be careful about what elements of a VLE I kept around after the end of the class. Possibly just completed work as a resources for subsequent classes, rather than formative discussions

Advantages:

• time and place independence

• no need to travel to the place of learning

• time lapse between messages allows for reflection

• speakers of other languages have added time to read and compose answers (and use translation tools if necessary)
  

• questions can be asked without waiting for a 'turn'

• it allows all students to have a voice without the need to fight for 'airtime', as in a face-to-face situation

• the lack of visual cues provides participants with a more equal footing

• many to many interaction may enhance peer learning

• answers to questions can be seen by all - and discussed.

• discussion is potentially richer than in a face-to-face classroom setting

• messages are archived centrally providing a database of interactions which can be revisited

• the process of learning becomes more visible to learners and tutors.

Disadvantages:

• communication takes place via written messages so learners with poor writing skills may be at a disadvantage

• paralinguistic cues (facial expression, intonation, gesture, body orientation) as to a speaker's intention are not available, except through combinations of keystrokes (emoticons) or the use of typeface emphasis (italics, bold, capital letters)

• time gaps within exchanges may affect the pace and rhythm of communications leading to a possible loss in textual coherence

• the medium is socially opaque; participants may not know who or how many people they may be addressing

• the normal repair strategies of face-to-face communication are not available and misunderstandings may be harder to overcome

• context and reference of messages may be unclear and misunderstandings may occur

• loss of spontaneity and quick-thinking and response type of debate / discussion.

The thing to note here is that these are not all advantages and disadvantages for the same party, and therefore there is an acess/equality dimension to this, and also a need for sensitivity to those differences.

this is quite important, and I think relates to the concept of 'work' up-post:

'If knowledge construction is the task, then the sharing of different perspectives through debate and dialogue is, within a social constructivist framework, a precondition for it but does not simply constitute it. Getting to the point of actually constructing new knowledge (new to individuals and maybe new in the public domain) requires skilled, sustained, purposive and directed analytic and creative conceptual communication practices' (Edwards, 2002).