Metrics & ROI

 

 

 

Internet Time Group helps companies get the most out of eLearning.

Our white papers

Here's more about what we do.

 

 

 

Power Shift

eLearning infrastructure decisions are climbing up the corporate ladder. A few years ago, eLearning was pigeonholed as a cheaper, faster way to train employees. By default, eLearning decisions fell to the director of training or HR.

Recently, functional managers have begun using eLearning to meet business objectives. Managers look beyond employees to customers, suppliers, and distribution channels -- everyone benefits from seeding eLearning throughout the value chain. This is where we are now, with eLearning decisions seesawing back and forth between can-do functional managers anxious to get on with it, and CIOs/CLOs who want to go the next step to enterprise solutions. Still rare but perhaps the next step in this evolution is the CEO who looks at eLearning as a competitive weapon, the way to create a nimble organization, improve customer service, move quickly, and stay ahead of the pack.

Systems Changes

Traditional ROI has suckered corporations into evaluating learning initiatives on a project-by-project basis, and this has lead to supporting each new approach as if it existed in isolation. The Meta-Learning Lab is developing ways to improve the overall learning process.

Take the old cliché of "Give a man to fish and he won't be hungry today. Teach a man to fish and he will never be hungry again." (Excuse the sexism; this dates back several thousand years.) The Meta-Learning Lab's goal is to teach fishermen how to improve their catch.

Scientific rigor: The Baloney Detection Kit

How to draw boundaries between science and pseudoscience, or between useful metrics and pure hype. From Scientific American

1. How reliable is the source of the claim?
2. Does this source often make similar claims?
3. Have the claims been verified by another source?
4. How does the claim fit with what we know about how the world works?
5. Has anyone gone out of the way to disprove the claim, or has only supportive evidence been sought?

6. Does the preponderance of evidence point to the claimant's conclusion or to a different one?
7. Is the claimant employing the accepted rules of reason and tools of research, or have these been abandoned in favor of others that lead to the desired conclusion?
8. Is the claimant providing an explanation for the observed phenomena or merely denying the existing explanation?
9. If the claimant proffers a new explanation, does it account for as many phenomena as the old explanation did?
10. Do the claimant's personal beliefs and biases drive the conclusions, or vice versa?

"Clearly, there are no foolproof methods of detecting baloney or drawing the boundary between science and pseudoscience. Yet there is a solution: science deals in fuzzy fractions of certainties and uncertainties, where evolution and big bang cosmology may be assigned a 0.9 probability of being true, and creationism and UFOs a 0.1 probability of being true. In between are borderland claims: we might assign superstring theory a 0.7 and cryonics a 0.2. In all cases, we remain open-minded and flexible, willing to reconsider our assessments as new evidence arises. This is, undeniably, what makes science so fleeting and frustrating to many people; it is, at the same time, what makes science the most glorious product of the human mind."


The ROI of eLearning

Jack Phillips, Ed Trolley, and Jay Cross discuss the ROI of eLearning. 9/2000

Jay Cross—Internet Time Group

Jay has been passionate about harnessing technology to improve adult learning since the 1960s. He has managed several software startups and is the former president of MegaMedia WorldWide. Jay founded Internet Time Group to help organizations learn. The group provides hands-on advice on launching new products, building reputation, and accelerating sales. Reach Jay at www.meta-time.com.

Jack J. Phillips, Ph.D—The Jack Phillips Center for Research

As a world-renowned expert on measurement and evaluation, Jack provides consulting services for Fortune 500 companies and is the author or editor of more than 30 books including The Consultant's Scorecard and HRD Trends Worldwide: Shared Solutions to Compete in a Global Economy. His 27 years of corporate experience in five industries led him to develop the ROI Process™—a revolutionary process that provides bottom-line figures and accountability for all types of training programs.

Edward A. Trolley—The Forum Corporation

Ed, Senior Vice President of The Forum Corporation, spent 26 years with DuPont, where he held line leadership positions in three different business units before being named Manager of Training and Education for the multinational corporation. While with DuPont, Trolley conceived the insourcing alliance concept "Running Training Like a Business." Since joining Forum in 1996, Trolley has carried the insourcing concept to leading global companies, working closely with the human resources leaders and line managers alike to significantly increase training's effectiveness and efficiency. Ed Trolley is at www.forum.com.


eLearning and Cost-effectiveness

In the June 2001 issue of T+D magazine, Samantha Chapman writes, "The fact is, as thousands of people around the world including myself and my boss learned the hard way, there are few claims as outlandish and as unsubstantiated as eLearning being a money saver." I couldn't resist sending this reply:

1. COSTS ARE NOT THE WHOLE STORY.

Costs are only part of the eLearning equation. The primary advantage my clients attribute to eLearning comes from increasing the top line, i.e. enabling organizations to do things they couldn't do with traditional training.

For example,

Transforming a traditional business into an eBusiness
Certifying thousands of employees in technical disciplines
Accelerating revenue with simultaneous global product launches
Upgrading customer service by building skills during non-peak hours
Merging organizations on Internet time
Providing scalable training to channel partners

The cost of eLearning is relative. So long as the bottom line is increasing, organizations shouldn't worry too much about costs.

2. IT'S eLEARNING + ILT, NOT eLEARNING *OR* ILT.

Your article suggests that eLearning is an either/or situation. This is a fallacy. Think of eLearning as supplementing traditional learning rather than replacing it. Only apply e-technologies when the benefits are obvious.

Take prework as an example. Traditionally, prework might take the form of reading material on paper. In today's fast-paced world, the material is often out-of-date before it's read. And whether anyone reads it -- and gets it -- before a workshop is anybody's guess. The eLearning alternative is to post the prework on the web, where it's always current, and require a passing pretest score for admission to the workshop.

3. THE PAST IS A SUNK COST.

When making an incremental decision, count only incremental costs. Most of my customers already have desktop computers, local area networks, and Internet access in place. The money already spent on these resources shouldn't be toted up as an expense of eLearning.

4. THE LARGEST COST OF ALL IS FOREGONE OPPORTUNITY.

Again and again, I've found the largest overall cost of any corporate learning endeavor is the cost of people's time. I'm not talking about salaries and benefits; I refer to the value they would have created had they not been tied up in training. Opportunity cost per hour is not a fixed amount. A salesperson's time during working hours in peak buying season is worth much more than the same individual's time after closing time in non-peak season. eLearning often enables the employee to shift learning to those non-peak hours.


Payback

Technology-enabled learning creates value by speeding things up. Business-school professors compare making big corporate changes to turning around the Queen Mary. Turn the rudder and in a few miles, the ship changes course. These days, organizations that lack the agility to turn on a dime can only go about as far as the Queen Mary (which is moored in cement alongside a pier in Long Beach, California.)

A Fortune 50 company used eLearning, knowledge management, and collaboration to bring new-hire sales people up to speed in six months instead of fifteen. Nine months x 1400 new hires/year x $5 million quota = $5 billion incremental revenue. To be sure, better products, sales campaigns, and a host of factors contributed to the gain but a tiny faction of $5 billion still yields a significant ROI. (Here are the details: New-hire training at Sun Microsystems.)

Ten thousand consultants at a Fortune 100 technical services company earned professional certifications via eLearning. The result? Less attrition, better esprit de corps, and $100 million revenue/year attributable to higher billing rates.

A software firm launches a new system into a $250 million global market with eLearning and virtual meetings. This accelerates time-to-market by two months, gives them first-mover advantage over a major competitor, builds a more confident and enthusiastic sales force, and gets the channel up to speed at the same time as the direct sales force. Gain? $80 to $100 million incremental revenue.

A very large retailer of personal computers realizes that customers are frustrated with their products because they don’t understand the software that accompanies them. The company offers customers free admission to an online learning community created by SmartForce. More than 100,000 customers sign up to learn Windows, Word, and Office apps online. Value of increased customer loyalty? Conservatively, $20 million in repeat business over three years.

Often an e-Learning initiative pays for itself right off the bat by eliminating travel and facility costs, but that misses the point, because in comparison, upside gains dwarf cost savings.


Internet Time Group White Papers

A Fresh Look at ROI -- ROI is relative. Where you stand depends upon where you sit.


Leveraging the People Value Chain

How to increase human capital, customer loyalty, and shareholder value on Internet time


Déja vu, 4/01/01

Return on Investment, The Measurement Conundrum, Institute for the Future, 11/99 [no longer on the web]

“Over the next few years, how well and how fast people learn will differentiate the successful from the unsuccessful companies. Past preoccupation with a narrow technical concept of return on investment (ROI) has interfered with the new way of thinking about organizational fitness. The learning effectiveness index of the future will emerge from integrating performance expectations with knowledge development.”

Déjà vu. Last week I was working with the Institute for the Future on what training and education may look like around the world circa 2007. My research led to a on the ROI of training. I was floored, for it felt as if the authors had been peering directly into my skull. These are my memes! I’ll intersperse excerpts from the article (in black) with my comments (in red).

For the past year, I’ve been ranting that knowledge will separate the winners from the losers, that traditional ROI measures the wrong stuff, and that business metrics are the only legitimate measure of the effectiveness of learning. See the transcript of the Return on Investment Roundtable.

“Traditional ROI models will continue to disappear in corporate and executive education and training. They have generally failed as measures of the effectiveness of education and training, except in some specific skills training.”

Sure. But often the education and training themselves have often failed, too. If your car’s gas gauge shows empty, you might have a broken gauge. On the other hand, you might be out of gas.

“Demand will rise for proof that education is a good investment, but measures and outcomes will be embedded in business results. Slowly, line managers will take over training and education, which will be aligned to business strategy, and effectiveness will be measured against results.”

At the end of 2000, astute executives were assessing the top-line strategic benefits of eLearning, e.g. transformation, entering new lines of business. In the current economic downturn, I’m seeing more emphasis on nuts-and-bolts cost-cutting and revenue generation. Either way, trainers are taking a back seat to line managers. See our recent article How to increase human capital, customer loyalty, and shareholder value on Internet time.

“ROI models work for particular skills when desired results are obvious. Trying to measure knowledge development the same way is the problem. Most companies – we estimate 75% -- have had so much difficulty that they have stopped trying to measure ROI.”

Yes! See our article A Fresh Look at ROI.

“It’s been clear for a long time that training for measurable skills is different than education for knowledge workers. Failure to recognize the split was one cause of the ROI conundrum.”

The term eLearning, popularized since the Institute’s article was written, has perpetuated the error of muddling skill (generally explicit) and knowledge (often tacit).

“The search for the Holy Grail of discrete measurement is fruitless for knowledge workers. Managers are realizing it’s better to spend nothing on ROI than to try to prove something you can’t measure.”

“It’s like email,” says Cisco’s Tom Kelly. “How much do you need?” You need enough to get the job done.

“Increasingly, training and education will be used as tools for employee recruitment and retention.”

Yes! See our white paper, Leveraging the People Value Chain.

“Businesses can no longer be historically focused, relying on old measures and indicators limited to what is already known and has been done. Business is changing fast. Training, education, and measurement models that seemed valid even a couple of years ago may be irrelevant now.”

Right on! See our screed, The eLearning Emperor Has No Clothes, further down this page.

“The new education model, the opposite of training someone to do what you want them to, helps people learn how to navigate for themselves. A focus on business outcomes will eliminate the quest for metrics to measure the “intangibles” of innovation, people skills, adaptation, and other qualities that fall into the “fuzzy-measurement” category.”


The eLearning Emperor Has No Clothes

Go to any major conference for trainers and you’ll find many sessions on evaluating results and measuring performance. If you’re a line manager with no training background, you will at first be confused when participants make statements like, “We evaluate 100% at Level 1, 80% and Level 2, and 40% at Level 3. We’re going to shoot for some Level 4 next year.”

The “levels” come from a taxonomy developed by a budding academic, Donald Kirkpatrick, as his Pd. D. thesis more than forty years ago. Level 1 evaluates trainee reaction (generally via evaluation forms derisively known as “smile sheets.”) Level 2 checks retention – can they pass the test? Level 3 looks at whether they can do what they were trained for – demonstration. Level 4 is whether the learning has created successful performance on the job.

Picture this. A national sales manager is reviewing quarterly sales performance with his boss. He tells her the new sales trainees scored 95% on Level 1, 92% on Level 2, and 89% on Level 3. Unfortunately, Level 4 improvement was infinitesimal. So the sales force loved the sales training, the majority passed the test, and nearly four out of five could demonstrate great sales behavior in a role-play. The only trouble is Level 4: they aren’t selling. How long would the sales manager keep her job? In business, Level 4 is, in fact, the only thing that matters. No wonder senior managers question the value of training.

The only valid measure of training is business metrics, not training metrics.


The ROI of e-Learning, Chapter 1

The E-Learning Revolution
 By Patricia A. Galagan

 Strategic at last

"The rise of e-learning is dividing training into two streams: corporate learning-usually owned by HR or corporate training departments-and training for a variety of strategic purposes, usually owned at a level just below the CEO. The strategy-linked learning programs may target people beyond the company such as partners, suppliers, and hoped-for customers. But companies are also using e-learning to change direction quickly, such as going from a traditional business to an e-business."

 

 

As the Godfather said, "This is business." If you can't see a benefit, don't do it.

 

Jack Zigon's list of performance measurement sites

 

Excerpt from Ed Trolley's Running Training Like a Business

 

Evaluating e-Learning
by Dorman Woodall

 

Jay's notes on making the business case, new ROI challenges

 

The trouble with the "four levels" is that they falter when they go outside of the limited context of training. What happens outside the box is what counts inside the box. See Measuring Training ROI & Impact (1999). You can guess how I see this.

 

The Evolution of Management Accounting
by Robert S. Kaplan

 

Show me the ROI, Inside Technology Training 11/99

 

Measuring ROI For The Top Line Of The Business
InfoWeek 4/99

 

BNH on ROI. Their software models simlify complex ROI calculations.

 

Discussion group: ROInet

The ROI of Human Capital

The Business Case website

The Fallacy of ROI Calculations

Measuring the Success of Training

Baruch Lev

Measuring the ROI of Training, CIO

Economic Value Added (EVA)

 

Learning at home

Training magazine, the March 2000 issue: Train on your own time, not "during work."

Sure, moving training from the classroom to the Web can mean reduced travel costs, less learning time away from the job, and certainly lower delivery costs. But most corporate training doesn’t require travel, says Paul Reali, president of CyberSkills Computer Training Centers in Winston Salem, NC. And, he points out, no valid study has yet shown that online delivery significantly reduces learning time—actual time spent mastering a skill or acquiring knowledge—compared with instructor-led training of similar quality.

“No one wants to tell you that the ‘anytime’ of online learning is supposed to be after work and that the ‘anyplace’ is at home,” he says.

Another reason: Despite yellow crime-scene tape barriers and “do not disturb” signs, the cubicle is a tough place to have a quality learning experience. And it’s almost impossible to reserve the necessary time and concentration without broad organizational support—and the backing of trainees’ immediate managers—for regular learning “time-outs.”

http://www.trainingmag.com/

This is so true and so short-sighted.
 

 

 


© 2003 Internet Time Group, Berkeley, California