Evaluating learning material to be acquired or developed

When deciding on a developer or a supplier of off-the-shelf SAQA-aligned outcomes-based learning material and assessments, the Training Provider's dilemma is how to evaluate the potential contractor or supplier.

Some time ago, the question arose of what guidelines can be given to training providers when they have to decide on contracting a designer/developer of outcomes-based learning material and assessments or when they buy 'off-the-shelf' material.

The suggestions that follow (which are by no means hard and fast rules or an attempt to be exhaustive) might be useful.

A little bit of homework may save you money and frustration down the line.

Although there is no fool-proof recipe, the following guidelines should be helpful:

  • Credibility - the qualifications, experience and subject matter expertise of the person who designed and developed the material. Here, qualifications do not only include 'technical' training in the topic, but also training in design and development of outcomes-based material and assessments. A developer with at least a few years of experience as Facilitator, Assessor and Moderator has a firm base for understanding what makes for material that meets education and training standards and expectations - from all points of view. Actual work experience in the field (e.g. supervisor/manager/sales assistant) provides a solid background in understanding the key deliverables for a target group; this will help in the development of material that helps learners acquire the skills they need.
  • Reputation - can the developer provide references that give evidence of reliability in terms of delivery? 
  • Validity and sufficiency of assessments to meet moderation requirements (See the separate post on validity and sufficiency of evidence).
  • Assurance. Find out what assurance the supplier is prepared to give, for example:
  1. Guarantee that material meets/will meet moderation and SETA verification
  2. Remedial action should it be necessary.

The criteria for assessments are that they should be:

  • Valid
  • Authentic
  • Current
  • Sufficient

Validity implies the assessment focuses on the requirements laid down in the unit standard, i.e. the assessment is fit for purpose.

Validity therefore requires the assessment to focus on what the specific outcome and related assessment criteria require the learner to be able to do. For example, if the assessment criterion states 'Explain..' the assessment should be designed in a manner that requires the learner to list, identify, etc. When the learner is required to 'Apply..' the assessment should be designed in a manner that requires the learner to provide evidence that the knowledge (such as 'using benefits to address customer needs') has been applied in a work environment. In the example of using benefits to address customer needs, the evidence will have to provide proof of this happening.

Sufficient implies that the evidence collected establishes that all criteria have been met and that performance to the required criteria can be repeated consistently. 

In my experience as moderator of learning programme material and assessments, the biggest shortcoming that I have observed in assessments designed by lesser-experienced developers of material and assessments is that there is lack of validity or misinterpretation of the specific outcomes and/or assessment criteria. For example, outcome 3 for US 118029 refers to hygiene practices in a wholesale/retail outlet (i.e. housekeeping and hygiene) and not merely personal hygiene of employees.

To ensure that the material developed meets the assessment criteria, it should be moderated before facilitation - a requirement of all SETAs. Such moderation will not only meet the SETA ETQA Quality Management System requirements, but might also save you the embarrassment and frustration that comes at the moderation stage if your material does not meet the requirements of the unit standard appropriately.

Views: 780

Add a Comment

You need to be a member of Skills-Universe to add comments!

Join Skills-Universe

Comment by Cobus Cato on August 10, 2012 at 8:37

Hi Marie

We agree once again, yet another concern I have is that many employers are lead by the nose by Training Providers who present them self as the experts (which they actually should be) because employers do not have the understanding of the Skills Development environment we have and this make our job much more difficult in giving guidance to employers.

Comment by Jackie Franck on August 7, 2012 at 8:23

Thank you Marie, that's very kind of you.  I've just handed in my Portfolio of Evidence after completing the training, and hope to get positive feedback.  It really is a challenging but stimulating function (design and develop).  I work for a retail organisation where our employees are our "clients", and have not yet started designing and developing course material, but will certainly be in touch should I require some assistance. I know Wilma Guest-Mouton, so will be able to get your details from her.  Thanks again for your offer of mentorship!

Comment by Marie Smith on August 6, 2012 at 18:06

Jackie, we all started with no experience. It might help to get a moderator/mentor to have a look at your material before you submit to a client. I am prepared to mentor should you wish.

Comment by Marie Smith on August 6, 2012 at 17:04

Wilma, it is a privilege to be part of the team of Guest Resource Services, where the outcome is quality education, with involvement of all stakeholders throughout the delivery process.

Comment by Wilma Guest-Mouton on August 6, 2012 at 16:30

Marie I am glad that you are part of our team. You know that one of our success recipes is to involve the trainers with our pre-moderation meetings, and although trainers are not always experienced enough, we do give them the opportunity to participate in order to enable them to grow as critical thinkers. I think all true educators strive towards quality education.

Comment by Marie Smith on August 4, 2012 at 13:04

Cobus, that is exactly my concern - not only the quality of the material but especially the assessments. Both learners and employers are jeopordised - learners are disillusioned when they realise they don't have the skills they were supposed to have acquired through training. Employers are also disappointed and they lose money spent on training and on the time that learners were away from the workplace. The sad issue is that training is losing its credibility. And in many cases it is Skills Development Levies that are wasted. If I was an employer I would be very selective in selection of training provider.

Comment by Cobus Cato on August 4, 2012 at 8:49

Marie, one often assumes that if the company, or developer/provider, is a reputable institution that all will be fine. I recently visited an internationally recognised organization to view their learning material. The material content was excellent but the assessments raised a concern with me. The training was unit standard training but the assessments did not address half of the assessment criteria of the specific outcomes. This is in my view a concern as one would assume that if the holder of a specific certificate of competence is trained and assessed against unit standards that the assessment criteria will be adeqautely addressed and competence/ability confirmed. I agree completely with the moderation of learning and assessment material before it is implemented.

Comment by Charles Dey on August 4, 2012 at 4:08

Referring to Sonja's query about what to charge, in my industry (International Supply Chain Management), for qualifications in the NQF 2 to 4 band, the giong rate is +/-  R2500 per credit. 

Comment by Jackie Franck on August 3, 2012 at 18:27


While I understand your comment about using developers with experience, there are some who would really like to get into this field and are not necessarily jumping on the bandwagon.  I've recently attended training on programme design and development, and during the training I realised that this is something I'd really like to do as a career and I believe I have the qualities to do so.  How do I gain the experience?  I also couldn't agree more with comments regarding credibility etc of material.

Comment by Marie Smith on August 3, 2012 at 17:28

Rolf, I am aware of more than one case where material approved by a SETA did not meet the specific outcomes and assessment criteria. It is obviously for Providers an additional cost to have the material moderated before facilitation. I don't know how diligently this aspect is audited during SETA verification visits for maintenance of a provider's registration but can only gather that there are some gaps. Providers should just know that not all SETAs do proper moderation for material approval (I have personally observed this process). The risk is that when it comes to post-assessment moderation the learners are the ones who are losing out if the internal moderator or SETA moderator does not endorse results at that last stage. I know of one instance where a whole learnership was completed where not one of the learners received their credits after the whole year of learning and effort.

© 2017   Created by Alan Hammond.   Powered by

Badges  |  Report an Issue  |  Terms of Service