applications, 113 

266

benefits, identifying intangible, 120 

communicating, 112 

data collection using, 31, 34 

defined, 108–109 

247

effectiveness of, 117 

follow-up, 118, 119 

background, 355–356 

how it works, 109, 110 

evaluation planning and methodology, 

integrate into ADDIE, 109, 111–112 

356–357, 364 

knowledge check, 122–123 

knowledge check, 371 

managers/supervisors, role of, 112 

objectives, 356, 358–363 

monetary value, determining, 114–117 

recommendations, 368, 370 

objectives, identifying, 109, 111 

results, 364–368, 369–370 

participants, 112 

peer/facilitator review of, 117–118 

reporting results, 120–122 

return-on-investment, calculating, 119–120, 

121

role of, 113 

teaching, 114 

time needed for, 114 

why use, 107–108 

statistics, 193, 196 

steps for, 194–195, 197 

techniques for, 191–195, 196 

audience, 245 

Management, 193 

case studies, value of, 241 

Chain of Evidence, 273–274 

credibility of program team, role of, 242 

analysis, 38–39, 41 

feedback, 248–249 

categories, 203–207 

format, 246, 247 

estimating, 207, 208–209 

implementation and, 288–289 

estimating implementation, 226–227 

importance of, 239–240 

estimating planning, 226 

knowledge check, 250–251 

fully loaded, 202 

media selection, 241, 246 

identifying, 201–211 

plan, 242–244 

importance of, 201–202 

reactions to, 250 

knowledge check, 210 

reasons for, 244–245 

pressure to disclose, 202 

results to management, 249 

prorated versus direct, 203 

scorecards, 246–248 

sources of, 203 

targeted, 241 

without benefits, 202 

testimonials, 242 

timely, 240 

unbiased, 242 

cut-off scores, 79–81 

benefits and pitfalls of, 12 

designing, 73–74 

needs assessment and, 13 

guidelines for creating, 82–83 

knowledge check, 81–82 

length of, 77–79 

model for, 74, 75 

validity, 76–77 

value of, 74 

benefits to, 228 

weighting items, 78–79 

databases, 193, 196 

estimating, 194, 195, 196 

experts, 193, 196 

guidelines, 198 

historical costs, 193, 196 

importance of, 189–191 

intangible measures, 196, 197 

advantages and disadvantages of, 39, 40–41

knowledge check, 198–199 

converting measures to monetary value,

standard values, 191–193, 196 

189–199

costs, identifying, 201–211 

implementation costs and benefits, 226–227 

costs/financial, 38–39, 41 

knowledge check, 234 

isolating effects, 38, 40, 173–187 

planning costs and benefits, 226 

key questions, 39, 41 

rationale for, 224 

planning, 35, 42–43 

return-on-investment, 226–232 

qualitative, 38, 40, 165–171 

statistical analysis, 232 

return-on-investment, calculating, 38–39, 

systems modeling, 232–233 

40–41, 213–222 

timing (continuum), 224–227 

statistics, 38, 40, 149–164 

tips, 233–234 

type to use, 147 

utility analysis, 233 

action planning, 31, 34, 107–124 

advantages and disadvantages of, 32, 33–35 

case studies, 32, 34–35 

criterion-referenced tests, 73–84 

See leadership development case example 

focus groups, 31, 34, 97–106 

interviews, 31, 33–34, 85–95 

key questions, 32 

application measures, 259–260 

multiple methods, using, 89 

impact and return-on-investment measures, 

performance records, 32, 35, 135–145 

260–262 

planning, 30 

importance of, 253, 254 

planning template, 32, 36–37 

influencing, 262–263 

success case method, 125–134 

investment level, 255 

surveys and questionnaires, 31, 33, 55–72 

knowledge check, 263 

tests, 31, 33 

learning measures, 258 

methods for obtaining, 254 

metrics and, 255–256 

reaction data, 257 

converting measures to monetary value, 

assessment of, 105 

194–195, 196 

benefits and pitfalls of, 12 

forecast analysis, 174, 175, 180–181 

conducting the session, 104 

data collection using, 31, 34 

facilitator, selecting, 98, 99 

group interview questions, 99–100 

importance of, 97–98 

Ishikawa diagrams, 100–101, 102 

knowledge checks, 105 

needs assessment and, 12 

nominal group technique, 101–104 

participants, selecting, 98 

session format, 98–99 

solutions, identifying, 105 

advocacy and support, 285 

checklist, 292 

communication, 288–289 

See also estimating future value 

experience, use of individual, 285–286 

experts, use of subject matter, 286–287 

goals, 284 

human resources and, 290–291 

identifying what to avoid, 284 

knowledge check, 293 

between current state and business 

outcomes, developing, 285 

requirements, 7 

performance improvement, 289–290 

plan, 292 

senior leadership, role of, 287–288 

background, 337–338 

technology and, 291 

business impact, 350 

business tools, 341, 343 

content and design of, 341, 342 

converting measures to monetary value, 

350–351 

advantages and disadvantages of, 11, 88 

isolation methods, 350 

assessment of, 93–94 

knowledge check, 353 

data collection using, 31, 33–34 

project team and steering committee, 341 

evaluation of training using, 87 

recommendations, 353 

executive, 254 

results, 345, 350 

focus group, 99–100 

return-on-investment methodology, 343–353 

guidelines for planning and conducting, 

scorecard, 352–353 

89–94 

value-based selling and pricing, 338–339 

knowledge check, 94 

needs assessment and, 10–11 

objectives, 89 

pilot testing, 92 

interviews and, 94 

isolating effects and, 185 

leadership development and, 335 

mLearning and, 318 

objectives and, 27 

performance records and, 143–144 

planning evaluations and, 50 

qualitative analysis and, 170 

questionnaires and surveys and, 70–71 

results and, 280–281 

return-on-investment, calculating and, 221 

simulation training and, 383 

statistical analysis and, 163–164 

Success Case Method and, 133 

technology and, 304 

background, 322–323 

benchmarking/best practices, 330–331 

knowledge check, 335 

multiple methods, 324–329 

outcomes defined, 323–324 

participant survey, 326–329 

results, presenting, 332–334 

See also mLearning 

action planning and, 122–123 

objectives, 20–21 

basic maintenance course and, 371 

opportunities for, viii–viv 

communication/reporting and, 250–251 

converting measures to monetary value and, 

198–199 

costs, identifying, 210 

criterion-referenced tests and, 81–82 

estimating future value and, 234 

executive viewpoints and, 263 

focus groups and, 105 

global program and, 353 

implementation and, 293 

benefits, xiv–xv 

challenges, vii–viii, xii–xiv 

application, 22, 23 

need for, xi 

constructing, 16, 18–19 

steps, xv–xvii 

impact, 22–23 

trends, viii 

importance of, 15 

input, 19–20 

interview, 89 

defined, 307–308 

knowledge check, 27 

Kirkpatrick levels, 309–310 

learning, 20–21 

knowledge check, 318 

levels of, 16, 17, 19–27 

role of, 308–309 

meaning and value of, 16 

traditional training versus, 310–316 

reaction, 20, 21 

return-on-investment, 23–24 

SMART, 16, 18 

summary of, 25 

V-model, 26 

monetary value 

converting measures to usable data, 142–143 

data collection and, 32, 35 

business requirement, defining, 6 

data sources, 141 

costs, 7, 8 

importance of, 135–136 

current state, identifying, 7 

knowledge check, 143–144 

gap analysis, 6 

linking with business measures, 136–140 

gaps between current state and business 

records, identifying appropriate, 140–141 

requirements, 7 

importance of, 3 

key questions, 4–5 

methods for, 10–13 

order taker versus the consultant, 4–5 

See also action planning 

solutions, identifying, 7–8 

evaluation project, 29–51 

steps, list of, 6 

importance of, 29–30 

training, value of, 8–10 

needs assessment, 3–14 

objectives, developing, 15–28 

tool, comprehensive, 44–48 

data analysis and, 38, 40 

importance of, 165–166 

knowledge check, 170 

quantitative versus, 166 

steps, 168–169 

techniques, 167–168 

validity and, 167, 168 

for business executives, 271–272, 273, 

276–277 

for business supervisors and managers, 271, 

accuracy, 64–65 

273, 275–276 

assessment of, 65 

Chain of Evidence, 273–274 

benefits and pitfalls of, 11 

communicating, 273–274 

body of, 60–61 

example of, 278–280 

conclusion of, 61 

for instructional designers and trainers, 269, 

content, creating, 57, 60–63 

273, 275 

data collection using, 31, 33 

Kirkpatrick levels, 266–267, 309–310 

distribution of, 65–70 

knowledge check, 280–281 

drafting, 63–64 

purpose of, 266 

effectiveness, optimizing, 63–65 

for training leaders and consultants, 

executive, 254 

269–271, 273 

feedback, 64 

for training participants, 268–269, 273, 274 

follow-up, 68–69 

types of, 267–268 

functionality, 64 

introduction to, 57, 60 

benefit-cost ratio, 214–215 

knowledge check, 70–71 

benefits, defining, 228–229 

needs assessment and, 10 

calculating, 38–39, 40–41, 119–120, 121, 

planning, 56–57, 58–60 

213–222 

purpose of, 55 

chain of impact, 220 

response rates, 65–66, 67 

converting benefits to monetary values, 228 

sample size, 65, 66 

costs, defining, 227 

scope of, 56–57 

defined, 215–216 

steps in developing, 55–56 

example of forecasting, 230–232 

structure of, 57 

forecasting, 226–232 

testing of, 64–65 

importance of, 213–214 

knowledge check, 222 

net present value, 217–218 

payback period, 216–217 

t-test, 162 

variation measures, 152–153 

programs suitable for, 220–221 

sensitivity analysis, 230 

targets, 218 

86–87 

applications, 125–126 

business case, 127–128 

knowledge check, 133 

mean or average effect, 128 

training realities, 127–133 

Training, 278–280 

decision-making tips, 296–297, 300–304 

(Aldrich), 374 

examples, 300 

implementation and, 291 

KnowledgeAdvisors, 298 

knowledge check, 304 

Learn.com, 299–300 

objectives, classifying, 301 

Sensei/ROI, 298–299 

SurveyMonkey, 297 

central tendency measures, 150–152 

Chi-square, 162 

effective versus effectiveness, 267–268 

converting measures to monetary value, 193, 

realities, 127–133 

196

success, 126–127 

data analysis and, 38, 40 

value of, 8–10 

descriptive, 150 

estimating future value, 232 

traditional 

importance of, 149–150 

data collection, 376 

inferential, 150 

differences in leader effectiveness, 382 

interpreting and understanding data, 

differences in perception of leadership, 

153–158 

379–381 

knowledge check, 163–164 

kurtosis (skewed data), 159–161 

knowledge check, 383 

outcomes, 376–377 

research method use, 375 

results, 377–378 

VL2007, 374–375 

86–87 

approaches to managing, 168 

qualitative analysis and, 167 

test, 76–77