Recruitment of National Assessment Resource Project Manager

Following a very successful meeting between SQA, LTS and the Scottish Government, we are now scoping a programme for the development of an itembank to serve the new Curriculum for Excellence.   A project manager will also be recruited to manage this development – for more details about this position please see

National Assessment Resource Project Manager

Metadata and vocabularies

Last Friday Becta came in to demonstrate its Vocabulary Bank for senior management in SQA in partnership with LTS, COLEG and Scran.The aim was to raise awareness internally of the importance of metadata and vocabularies in an electronic environment, demonstrate the tools available to manage them and highlight what other organisations are doing in this area.

We also looked at how we could link in with a potential future itembank, CMS or repository and this is possible quite easily via the webservice SRU. LTS have already demonstrated this with linking the bank to their own CMS. This means we can potentially create and manage our own taxonomies as well as integrate them with the metadata input mechanism for authors.

The partnership work with Becta, LTS, COLEG and Scran has been a good driver for work in this area internally in SQA and we’ll keep you up to date with the developments….

Presentation on Avatar Signing

Also at the CAA conference, we gave our first presentation on avatar signing to the CAA community.  The feedback was really positive and several people had lots of ideas on how our new technology can be used.   The current project comes to an end pretty soon and we are currently considering how we can build on this research work to maintain our presence in this new and exciting area.

Powerpoint presentation: avatar-signing-in-examinations.ppt

Short paper: incorporating-avatar-signing-into-assessment-items.doc

FREMA – Domain Referencing for Assessment

FREMA was a project based at Southampton University which tried to scope the assessment domain – breaking it up into seperate services and examining how people would use service orientated assessment archetecture.  It ran from April 2005 to October 2006 and interviewed many assessment practitioners to gather usecases and explore the different aspects of the assessment domain.

 The most useful outcome IMHO were the concept maps that were constructed to try to break up the domain and impose a bit of structure on an area with many disparate users.  With new services being developed all the time in the assessment domain, such an approach can ensure that service development doesnt suffer from feature creep, where a service which starts out to fulfill one service requirement ends up doing two or more, undermining the modularity of the SOA approach.

One concern that people had about FREMA however, was that although the outcomes were brilliant, without ongoing funding to ensure that new services were tracked as they were developed and to remind people about the framework, there was a danger that FREMA could become yet another idea that everyone thought was great, but slowly got eroded and forgotten about.  Now the bods at Southampton have turned the reference model into a semantic wiki.  The idea is brilliant, but its still a bit hard to use, even if you are quite familiar with wikis. 

At the CAA Conference  however,  they announced that they were planning on building a front end on to the wiki, which would make it a bit more intuitive to use, and hopefully ensure that the community could sustain the model under its own steam.  This would be a brilliant idea.   FREMA was too good a project to lose the benefits of, and just a little bit of buy-in could go a long way.

Usecases from the SPAID project

The SPAID project was an itembanking project that developed webservices related to itembanking.It was funded by JISC  , under the toolkits programme of the ELF.  Its particular focus, was on itembanking although it concentrated on some of the essential pre-requisites. In particular it looked at the metadata tagging, content packaging and usage data generation in items.  Here are some of the use cases that the project generated.

Metadata UseCases
Create Metadata 
Edit Metadata
Verify metadata
Auto-update Metadata

Content Packaging
Content Packaging Service
Auto-content Package

Usage Data Generation
Generate Usage Data
Update Usage Data

Itembanking with the Rasch Model

I attended a couple of seminars on Rasch at Cambridge Assessment in the past two weeks, one was an introduction to Rasch and the other a bit more detailed insight into Equating and itembanking with the Rasch model. These were very useful in developing an understanding of the underlying calculations necessary to implement this model. During the course we explored the implications of adding items to the bank (especially at the beginning of populating), how to link these with anchor items and in turn putting them on a common scale of difficulty. We also touched on item characteristic curves and had a demonstration of the RUM software.

For any further information on this course please visit: www.assessnet.org.uk

Item Analysis Project

In an itembanking system item analysis is important for monitoring candidate performance on items which often allows us to enhance the validity and reliability of these items. There are a number of item analysis theories and depending on the context in which the items are used, one may be more suitable than the other. The 3 most common theories are: Classical, Rasch and Item Response Theory (IRT). Both Rasch and IRT are forms of the Latent Trait Theory.

SQA currently uses Classical analysis for pre-testing objective items and this theory has been perfectly adequate for this purpose. However, as we are moving to an itembanking model there is a need to review our current model. The CAA unit has therefore undertaken an initial evaluation exercise in collaboration with the Research and Information Services (RIS) unit to determine which item analysis theory is most suitable for our data and for itembanking in general. We ran an analysis on our 2005 performance data and compared the outputs of the Classical theory and Rasch which is representing the Latent Trait theory. The findings suggest that although there is little difference in the outcomes, Rasch is the preferred option for an itembanking system as it allows item-independant analysis. More on the Rasch model soon…

Avatar Signing Trials

Our avatar signing trials took place earlier this week in Dingwall Academy, Donaldsons College and Deaf Connections in Glasgow.  Many thanks to all of the techncial and teaching staff who made these trials possible.

 Anna, our avatar, (right) has been working very hard and is now enjoying a well deserved rest!

Almost 20 people from teenagers to senior citizens took part in the trials and gave us their impressions of this work.  The results of the focus group are being translated from BSL into English and a report will be available at the end of the summer.

Metadata and Vocabularies

Metadata, vocabularies and taxonomies are essential in describing learning and assessment resources for searching, distribution and itembanking purposes and the more we can agree on them the more effective they become. 

Therefore, an interagency group was established in November 2006 with the aim to standardise vocabularies for a Scottish education context. The Joint Vocabularies Group is made up of representatives from SQA, LTS, COLEG, BECTA and SCRAN. It operates at two levels: the Advisory group and the Working group. The Working group has been looking at mapping and converging vocabularies from each organisation wherever possible, based on the SCORM implementation of the IEEE LOM. We plan to upload the finalised vocabularies to the BECTA Vocab Bank later in the year where they will be publicly available. I’ll keep you up to date with the developments!

5-14 Itembanking Updates

We have been working hard with Learning and Teaching Scotland, aiming to finalise the specification for the new 5-14 itembank, due to be available next academic year.  Specifications are now being finalised and the procurement process will begin within a few weeks.