Evaluation Tip 3: Ways to Gather Data

One of the most important aspects of conducting an evaluation is choosing the right ways to find information. There are questions you can ask before starting your evaluation process to help you chose the methods that are best for your situation.

Questions:

Who is interested in the evaluation results?

• Trainer?  Manager?  Organization?  Government Agency?

What questions do they want answered?

• Are skills/knowledge gained?  Did transfer occur?  Are there improvements?

What resources are available?

• Financial?  Time?  Personnel?  Equipment?  Materials?

There are many ways to gather information. Each has advantages and disadvantages.

Data Gathering Techniques

Questionnaires

• Advantage: Allows evaluators to quickly gather data from large groups.


• Disadvantage: Not always accurate because of factors such as people not responding honestly or accurately. Some reason for this are an uncomfortable testing environment, the desire to respond in a socially acceptable manner, and misunderstanding the instructions or questions.

Interviews

• Advantage: Allows the evaluator to gather data that is more accurate than that from questionnaires since the interviewer can verbally address any misunderstandings or questions. Can also ask for more in-depth information than is practical in written surveys.


• Disadvantage: Can be time-consuming and expensive if many questions are asked of many trainees. Analysis of in-depth data also takes a lot of time.

Facial expressions/Body language

• Advantage: Allows the evaluator to gather information without being intrusive.


• Disadvantage: One person's perception of an expression may not be the same as another.

Performance tests

• Advantage: Can measure the skills of a worker in a real or simulated work environment.


• Disadvantage: Can be difficult to simulate a work environment. If the test is conducted in the actual work environment, then it must be scheduled with regard to production concerns.

Written tests

• Advantage: Often standardized and validated before use.  A reliable, valid test is able to consistently measure the same thing every time it is used.  Written tests are usually completed in a classroom setting where large groups can be evaluated at the same time.


• Disadvantage: If there are problems with the testing environment (i.e., the room is too hot, the chairs are uncomfortable), the test-takers may become distracted and not respond accurately. Literacy or language problems can also be an issue.

Workplace observations

• Advantage: Gives clearest data about whether or not training is being used in workplace.


• Disadvantage: Requires evaluation some time after training in the work environment. This may interfere with production.

Team Games

• Advantage: Can be a creative way to engage individuals and keep their attention.


• Disadvantage: Games make it difficult to "measure" or evaluate individual trainees.

Group discussion

• Advantage: Can be a great way to gather information about training or to answer questions by creating an open forum where individuals can interact and talk.


• Disadvantage: Individual differences exist between those that participate in the discussion, and this factor may influence the type of information received.  If some individuals are quieter than others, feel pressure to conform to what others are saying, or are disinterested, they may not share information or report how they really feel about the training.

Analysis of statistics

• Advantage: The use of numbers and statistics is highly regarded in providing and understanding information.


• Disadvantage: The numbers can sometimes be manipulated in such a way that data can be misleading.  Misleading data can lead to incorrect or inaccurate beliefs about the information gathered regarding training.

If you are clear about the evaluation questions you need to address, you will be able to assess the pros and cons of using various data gathering methods and decide which are best for you.

For additional information contact: Launa Mallett, 412-386-6658, LMallett@CDC.GOV