Evaluating learning material to be acquired or developed 19


When deciding on a developer or a supplier of off-the-shelf SAQA-aligned outcomes-based learning material and assessments, the Training Provider’s dilemma is how to evaluate the potential contractor or supplier.

Some time ago, the question arose of what guidelines can be given to training providers when they have to decide on contracting a designer/developer of outcomes-based learning material and assessments or when they buy ‘off-the-shelf’ material.

The suggestions that follow (which are by no means hard and fast rules or an attempt to be exhaustive) might be useful.

A little bit of homework may save you money and frustration down the line.

Although there is no fool-proof recipe, the following guidelines should be helpful:

  • Credibility – the qualifications, experience and subject matter expertise of the person who designed and developed the material. Here, qualifications do not only include ‘technical’ training in the topic, but also training in design and development of outcomes-based material and assessments. A developer with at least a few years of experience as Facilitator, Assessor and Moderator has a firm base for understanding what makes for material that meets education and training standards and expectations – from all points of view. Actual work experience in the field (e.g. supervisor/manager/sales assistant) provides a solid background in understanding the key deliverables for a target group; this will help in the development of material that helps learners acquire the skills they need.
  • Reputation – can the developer provide references that give evidence of reliability in terms of delivery? 
  • Validity and sufficiency of assessments to meet moderation requirements (See the separate post on validity and sufficiency of evidence).
  • Assurance. Find out what assurance the supplier is prepared to give, for example:
  1. Guarantee that material meets/will meet moderation and SETA verification
  2. Remedial action should it be necessary.

The criteria for assessments are that they should be:

  • Valid
  • Authentic
  • Current
  • Sufficient

Validity implies the assessment focuses on the requirements laid down in the unit standard, i.e. the assessment is fit for purpose.

Validity therefore requires the assessment to focus on what the specific outcome and related assessment criteria require the learner to be able to do. For example, if the assessment criterion states ‘Explain..’ the assessment should be designed in a manner that requires the learner to list, identify, etc. When the learner is required to ‘Apply..’ the assessment should be designed in a manner that requires the learner to provide evidence that the knowledge (such as ‘using benefits to address customer needs’) has been applied in a work environment. In the example of using benefits to address customer needs, the evidence will have to provide proof of this happening.

Sufficient implies that the evidence collected establishes that all criteria have been met and that performance to the required criteria can be repeated consistently. 

In my experience as moderator of learning programme material and assessments, the biggest shortcoming that I have observed in assessments designed by lesser-experienced developers of material and assessments is that there is lack of validity or misinterpretation of the specific outcomes and/or assessment criteria. For example, outcome 3 for US 118029 refers to hygiene practices in a wholesale/retail outlet (i.e. housekeeping and hygiene) and not merely personal hygiene of employees.

To ensure that the material developed meets the assessment criteria, it should be moderated before facilitation – a requirement of all SETAs. Such moderation will not only meet the SETA ETQA Quality Management System requirements, but might also save you the embarrassment and frustration that comes at the moderation stage if your material does not meet the requirements of the unit standard appropriately.

Share on Social Media

Leave a comment

19 thoughts on “Evaluating learning material to be acquired or developed

  • Cobus Cato

    Hi Marie

    We agree once again, yet another concern I have is that many employers are lead by the nose by Training Providers who present them self as the experts (which they actually should be) because employers do not have the understanding of the Skills Development environment we have and this make our job much more difficult in giving guidance to employers.

  • Jackie Franck

    Thank you Marie, that’s very kind of you.  I’ve just handed in my Portfolio of Evidence after completing the training, and hope to get positive feedback.  It really is a challenging but stimulating function (design and develop).  I work for a retail organisation where our employees are our “clients”, and have not yet started designing and developing course material, but will certainly be in touch should I require some assistance. I know Wilma Guest-Mouton, so will be able to get your details from her.  Thanks again for your offer of mentorship!

  • wilmaguest-mouton

    Marie I am glad that you are part of our team. You know that one of our success recipes is to involve the trainers with our pre-moderation meetings, and although trainers are not always experienced enough, we do give them the opportunity to participate in order to enable them to grow as critical thinkers. I think all true educators strive towards quality education.

  • Marie Smith Post author

    Cobus, that is exactly my concern – not only the quality of the material but especially the assessments. Both learners and employers are jeopordised – learners are disillusioned when they realise they don’t have the skills they were supposed to have acquired through training. Employers are also disappointed and they lose money spent on training and on the time that learners were away from the workplace. The sad issue is that training is losing its credibility. And in many cases it is Skills Development Levies that are wasted. If I was an employer I would be very selective in selection of training provider.

  • Cobus Cato

    Marie, one often assumes that if the company, or developer/provider, is a reputable institution that all will be fine. I recently visited an internationally recognised organization to view their learning material. The material content was excellent but the assessments raised a concern with me. The training was unit standard training but the assessments did not address half of the assessment criteria of the specific outcomes. This is in my view a concern as one would assume that if the holder of a specific certificate of competence is trained and assessed against unit standards that the assessment criteria will be adeqautely addressed and competence/ability confirmed. I agree completely with the moderation of learning and assessment material before it is implemented.

  • Jackie Franck

    Tass,

    While I understand your comment about using developers with experience, there are some who would really like to get into this field and are not necessarily jumping on the bandwagon.  I’ve recently attended training on programme design and development, and during the training I realised that this is something I’d really like to do as a career and I believe I have the qualities to do so.  How do I gain the experience?  I also couldn’t agree more with comments regarding credibility etc of material.

  • Marie Smith Post author

    Rolf, I am aware of more than one case where material approved by a SETA did not meet the specific outcomes and assessment criteria. It is obviously for Providers an additional cost to have the material moderated before facilitation. I don’t know how diligently this aspect is audited during SETA verification visits for maintenance of a provider’s registration but can only gather that there are some gaps. Providers should just know that not all SETAs do proper moderation for material approval (I have personally observed this process). The risk is that when it comes to post-assessment moderation the learners are the ones who are losing out if the internal moderator or SETA moderator does not endorse results at that last stage. I know of one instance where a whole learnership was completed where not one of the learners received their credits after the whole year of learning and effort.

  • Rolf Kühnast

    “…it should be moderated before facilitation – a requirement of all SETAs.”
    I was recently contracted to a national FET college to facilitate a registered National Qualification at NQF4. On starting the course in June, there was no Assessment Guide, the activities in the learners PoE were not referenced to the S/Os of the Unit Standards, and additionally the activities in many cases were well beyond NQF 4.
    On raising this with the head of the department responsible for this qualification, I was told that the activities would only be moderated next year, and that the responsible SETA was satisfied with the documentation provided.
    I questioned how the activities were to be assessed,and against which Assessment Criteria: I am still waiting for a reply.

  • Hannes Nel

    As a private intitution with seven researchers of whom five have PhDs and an average of twenty years expereince in learning programme design and development we have a serious problem with evaluators who do not know what contextualisation means. They often use this as an excuse for not endorsing materials, but when you ask them what they mean it quickly becomes clear that they don’t understand. I will write an article on contextualisation of learning materials over the weekend and share it with you next week. Dr Hannes Nel, MD Mentornet

  • Sonja Broschk

    This query does not totally pertain to the topic at hand, but where would I find information pertaining to the cost of developing materials.  I have to quote for two documents that I am setting up and have no idea what to charge.  Can someone point me in the right direction, please?

  • Tass Schwab

    So many companies have run into disasters regarding training material – it is important to have a number of choices regarding the quotes as well as the sample of the training material. If someone says that they have developed for X company, phone and verify! I am an experienced developer and my skills are precious, it irks me that anyone thinks that they can jump onto the development bandwagon, rather use people who have some years under the belt regarding this! Thanks for this article!