It didn't take long for the test center proctor to
realize something was amiss. One group of people clearly stood out from
the rest of the candidates taking a popular IT certification exam. They
sat rigidly in their chairs, hardly moving at all, and they proceeded
through the questions at a pace of six items per minute, well above the
norm of one to two questions per minute. All scored well above the
minimum needed to pass the test.
After the testing concluded, the test center called in
Caveon LLC,
a consultancy that specializes in test security, including data
forensics, to review the situation. "At first blush it looks like by
using a Bluetooth speaker and a video camera they were collaborating
with a subject-matter expert offsite," says Caveon's vice president
Steve Addicott.
Such equipment is readily available online at sites like the aptly named
spycheatstuff.com.
Aspiring cheaters can buy wireless speakers that fit deep inside the
ear canal, where they can't easily be seen, as well as tiny cameras that
are simple to hide. The suspected cheaters in this case were most
likely sitting still to give their hidden cameras a clear video image of
the screen, Addicott says. The review of that particular case is still
ongoing.
Cheating is trending
IT certifications have become a primary route to both salary premiums and career advancement, according to a recent
Foote Partners report.
So it's no surprise that, as the popularity of certifications has
grown, so has cheating. "Jobs and careers are at stake here, so people
will attempt all sorts of things," says Matthew Poyiadgi, vice president
of Pearson Education Inc.'s Pearson VUE business unit, which manages
5,100 test centers worldwide and counts the IT certification program
manager CompTIA among its clients.
And while
CompTIA estimates that the level of cheating on IT certification exams
is less than 5%, industry insiders say the problem is growing and that
keeping up with the cheats requires constant vigilance.
How people cheat
- Bring high-tech spy cameras and Bluetooth earpieces into test
centers to show questions to and receive answers from an off-site expert
- Purchase stolen test content from overseas "brain dump" sites and then memorize the questions and/or answers
- Share questions and answers in online chat rooms
- Hire an expert as a proxy to take the test for them
- Bring low-tech cheat sheets into the test center on index cards, write answers on the palm of the hand, etc.
- Surreptitiously use a smartphone to gain unfair advantage through use of texting, images, online searches, etc., during an exam.
-- Robert L. Mitchell
So
far, cheating doesn't appear to have devalued most IT certifications in
the eyes of hiring managers. For the 309 IT certifications that Foote
Partners tracks, the average pay premium across 2,600 surveyed companies
has gone up for the last four consecutive quarters, says CEO David
Foote.
While there's no way to definitively know
if a prospective hire has cheated to obtain an IT certification,
employers can and should check with the certification body to make sure
the person actually attained it. "Trust, but verify," says Addicott.
For
the most part, he adds, hiring managers can trust that verified IT
certifications were legitimately earned."Just a few rotten apples have
cast doubt on the qualifications of individuals in the IT profession,"
he says. But, he adds, it is possible that a few individuals have
benefitted from the live exam content available online and used that to
gain a higher score on an exam. So an IT certification should only be
one part of the hiring decision.
Other steps
include checking references, reviewing employment history and asking a
few carefully crafted questions designed to gauge whether the candidate
really knows his or her stuff.
Where the cert developers fit in
Developers of IT certification programs, such as
Microsoft
and CompTIA, contract with Prometric, Pearson VUE and other independent
test centers that administer and proctor tests worldwide on their
behalf. These businesses also provide training services, and so must
have a secure firewall between the testing and training sides of the
business.
IT certification bodies and test
center operators are engaged in an arms race with pirates who steal test
questions and answers, and with cheaters who buy that information,
share answers in chat rooms, pay "proxies" (people who will to take
tests for them) and bring a range of technologies and techniques into
test centers to gain an edge. IT certification organizations, worried
about degradation of their credentials, are striking back by turning to
more sophisticated methods to catch cheaters and mitigate piracy. And
cheaters who get caught increasingly face more than just a slap on the
wrist.
Even people who cheat and don't get
caught during the exam still have reason to worry. Pearson VUE records
every session to digital video and reviews it after the fact. Recently,
scrutiny of unusual head movements tipped off the team that one test
taker had an embedded camera in his glasses. "The way people are
cheating is changing. They're using technology more," Poyiadgi says.
But
the most common ways people try to cheat aren't always the most
high-tech, says Shelby Grieve, Microsoft's director of professional
certifications including the Microsoft Certified Solutions Expert and
Microsoft Technology Associate. "The trend has moved from taking exam
answers into a testing center to more passive methods of cheating, such
as using 'brain-dump' sites and proxy testing services," she says.
Grieve
says Microsoft has caught candidates who colluded online through
question and answer sharing, as well as people who used low-tech
approaches such as copying off other peoples' exams, texting answers and
even modifying someone else's printed score report.
Potential consequences for those who get caught
- Immediate disqualification from the current test
- Ban on taking the test again for a period of time -- or for life
- Loss of all previous certifications from the IT certification program provider
-- Robert L. Mitchell
Brain-dump
sites don't just provide a place where users can share answers, says
Caveon's Addicott. "These websites aggressively sell pirated test
content and package it as test prep materials -- and they guarantee that
you'll pass. It's a real problem with IT certifications," he says. Most
of these sites are based in Asia, where it's more difficult to shut
down the sites and prosecute the offenders. Overseas test center
franchises with lax controls have been a source for test theft and
cheating because tests and answer keys are typically always downloaded
and stored at each location, giving cheaters easier access, he adds.
"The
single biggest factor in how much cheating you have is if you test
internationally, and IT certification programs are virtually all
international programs," says John Fremer, president of Caveon's
Consulting Services group.
Rise of the hired gun
Proxy
test-taking is growing concern for Bryan Kainrath, vice president for
certification operations at CompTIA, which owns the A+, Network+ and
other popular IT certifications. "We're seeing more proxy testing than
we have in the past. Most proxy scams involve hiring someone in China to
take a test for someone in the U.S. That happens all the time," he
says.
A few years ago, a large IT certification
provider engaged Caveon to hire a proxy and attempt to pass the test
without being caught. "The certification program paid us, we paid a
proxy service and one of my colleagues earned this prestigious
certification even though he had no background," says Addicott. The
price to cheat: A $1,000 check wired through Western Union. The terms
were 50% down, with the balance paid after the job was completed.
Proxy
test-taking services are big business overseas, in part because what
Americans consider cheating is culturally more acceptable in some other
locations, Caveon's Fremer says. The buyer signs up and the proxy goes
to a test center and takes the test. It's good money, says Fremer. "In
some parts of the world you can earn six months' salary with one proxy
test-taking event."
A sample letter from Caveon
LLC's interaction with a proxy website. By paying the site to hire a
proxy to take the test in his place, a Caveon staff person "earned" a
prestigious IT certification for which he had no background. Caveon
removed the name of the test to protect the client. Source: Caveon LLC.
In
some cases, proxies have been able to skirt security protocols by
visiting corrupt testing facilities overseas that operate both a
legitimate "front room" test area and a fraudulent "back room"
operation. "Those stringent protocols aren't followed when the test
center runs its own proxy ring," which can be very lucrative, Addicott
says.
To address proxy test-taking, test centers
typically require candidates to present a photo ID, and a few centers,
including those directly managed by Pearson VUE, have added biometric
identification and digital signatures, as well as taking the candidate's
photo. Once a person has registered under one identity, he can't act as
a proxy for someone else. What's more, the person who hired the proxy
will be caught if she tries to take another test, since her photo and
biometric data won't match.
Test centers might
also record the test subjects on digital video, and put the test taker's
photo right on the certification report. "Proxy testing used to be a
big thing," says Pearson's Poyiadgi. "But once we required digital
photos and digital signatures it disappeared."
But
while the "gold standard" of testing security applies to the 500
testing centers that Pearson VUE owns, that can vary at the other 4,600
sites owned by Pearson's partners, including IT training organizations
and colleges and universities that test students at the end of a
training program.
Den of thieves
Pirates
use a variety of techniques to steal entire tests and answer keys.
These include sending people into test centers to remember or photograph
sets of questions. (This type of "item harvesting" might require
sending as few as 10 people into a test center to memorize all of the
questions on a given test.)
It can also involve
outright theft of test data from corrupt or lax test centers. "Because
the whole test and answer key is downloaded to servers at each location
the entire item bank and answer key are available to be hacked. It's
really problematic," Caveon's Addicott says -- and it's leading some
certification and testing organizations to move to a SaaS-based test
delivery model. (
See sidebar, below.)
When
test takers try to cheat using brain-dump sites, however, they
sometimes end up getting cheated themselves. In some cases the sites
deliver fraudulent or obsolete content to unsuspecting buyers, says Dave
Meissner, chief operating officer at Kryterion Inc., a provider of
online IT certification testing services. "If people spent the same
energy and creativity to study as they do to cheat they would be far
better off."
In response, IT certification
bodies have staged coordinated attacks on brain-dump sites where the
pirates attempt to sell the looted data, including the use of cease and
desist orders and raids, says Kainrath. "We'll meet with
Cisco, Microsoft, VMware and try to figure out the best approach to mitigate these sites," he says.
"If
we find out that a test center has been colluding in any way, that
center is shut down by our security team," says Poyiadgi. Pearson VUE,
he adds, has only experienced "a handful of cases."
For
the industry as a whole, however, combating intellectual property theft
has been an uphill battle. "You can shut the sites down but it's like
pulling the top off a weed. It just pops up somewhere else," Kainrath
adds.
"It's not mom and pop" thieves, says
Fremer. "Organized sophisticated stealers can make millions -- or tens
of millions -- from just one certification program."
So,
test sites and certification programs try to react quickly to minimize
the damage. CompTIA monitors online brain-dump sites and chat rooms for
stolen test items, and uses analytics to determine whether any given
question's effectiveness in measuring competency might have been
compromised. "As soon as there's been any degradation we pull the item,"
Kainrath says. "We have huge item banks in reserve and can move
questions in and out quickly."
Story continues
on next page.
Attacking pirates from the cloud
The
traditional computer-based testing approach of having full copies of IT
certification tests and answers stored in thousands of test centers
worldwide has made test theft difficult to stop. To reduce the risk, IT
certification providers are beginning to adopt Internet-based technology
(IBT),
cloud-based software as a service methodology that delivers questions, one at a time, in encrypted form, to a secured
browser on each test taker's desktop.
This
approach eliminates the need to download and store tests and answer
keys at each testing site, which can have different levels of security
depending on their size and where they're located. "The use of IBT is
still relatively small but growing," Caveon's vice president Steve
Addicott says, and big players such as Microsoft and CompTIA are already
starting to adopt it.
At Microsoft, "We use the
traditional delivery engine as well as just-in-time, Internet-based
delivery," says Shelby Grieve, Microsoft's director of professional
certifications.
Internet Testing Systems LLC
sells software and online proctoring services that IT certification
programs and test centers can use via a private-label portal to deliver
content over the Internet to test takers anywhere. "We stream encrypted
test items one at a time and only decrypt them when rendered on the
screen," says Cabell Greenwood, vice president of business development.
Kryterion
offers IBT and online proctoring for IT certification programs. With
online proctoring, "There's no opportunity for any level of collusion
between the proctor and the test taker," says Dave Meissner, chief
operating officer at Kryterion Inc.
CompTIA is
working with Pearson VUE to deploy IBT, possibly later this year, and
Bryan Kainrath, vice president for certification operations at CompTIA,
is bullish on the technology's prospects. "We don't have to send the
answer keys. We pull the items back, take it offline, do the scoring and
send the results to the candidate. We can secure items for a lot
longer."
But IBT isn't always a good fit. It
requires significant bandwidth, and some testing centers, particularly
in overseas locations where the most intellectual property theft occurs,
don't have enough to reliably deliver tests in that way, Addicott says.
-- Robert L. Mitchell
That
process can present an expensive challenge, however, because organized
theft rings can compromise entire tests within three to five weeks of
when they're first released, while most IT certification exams are
refreshed every 12 to 15 months, Addicott says.
Kainraith
admits that's a problem, but he thinks that questions take a bit longer
to appear on brain-dump sites, and says CompTIA replaces tests at a
rapid pace. "We're able to churn our items a lot faster than 12 to 15
months," he says, although he declined to say how fast.
While
CompTIA has the scale and resources to turn over its test questions
more quickly, smaller IT certification programs are more limited because
the cost of building and maintaining tests ranges from hundreds of
dollars per question to thousands of dollars per test item, according to
Caveon.
Countermeasures: Tripping up the cheats
Catching
cheaters has become its own science. "More candidates are sharing
knowledge than we've seen in the past," says Kainrath. But both test
centers and IT certification owners have ways of figuring out who's
using stolen and shared test data, as well as who might be coming in to
steal it.
In addition to using live proctors,
Microsoft and others are moving toward online proctoring, which combines
the use of a video camera with a live feed of the test taker's screen.
While an online proctor is limited by what he can see on a video camera,
it's easier to take immediate action against cheaters, Grieve says.
Because they can look for suspicious activity at the question level,
online proctors can identify cheating sooner and end the test before the
candidate can see -- and possibly compromise -- the rest of the exam
content.
How test centers stop cheaters
- Use live and/or online proctors trained to spot suspicious activity
- Ban all electronic devices from the test room
- Perform forensic analysis of the test results to detect "anomalous" behavior that might indicate cheating
- Use
"Trojan horse" questions or other innovative test designs that tip off
test program managers that the candidate studied stolen test content
- Identity
validation with photo ID, digital signatures, biometrics; photograph
the subject and include it on the test report to thwart proxy test
takers
- Randomize order of multiple-choice test questions and answers
- Use multiple exam versions containing completely different questions
- Use
scenario-based questions that require that the candidate perform an
action by interacting with a simulation, rather than answering a
multiple choice question
- Use
adaptive testing that varies each successive question based on the
answer given to the previous one and stops the test as soon as
proficiency is determined
-- Robert L. Mitchell
Test
centers also have ways to tell if candidates have been memorizing
stolen test questions and answers or sharing knowledge in chat rooms.
"We leverage several different publication strategies and question types
designed specifically to address cheating," Grieve says.
While
Grieve declined to provide details, Addicott says some of the more
basic anomalies include people who perform at "superhuman speeds" on the
exam or who perform well on items that have been on the test a long
time while scoring poorly on newer items -- an indicator that the
individual may have memorized stolen test content.
Some
IT certification exams also catch people who have memorized stolen test
data by including "Trojan Horse" questions that deliberately include
the wrong answer in the official answer keys. These questions don't
count toward the candidate's overall score, but if the test taker
answers a predetermined number of such questions with the incorrect
answers listed in the answer key it's assumed that they used stolen
information and the test is automatically invalidated, says Addicott.
Certification
programs may also use different test designs in an attempt to thwart
cheaters who have memorized test questions and answers. These include
scrambling the order of questions on any given exam, randomizing the
order of answers to multiple-choice questions, having a pool of
questions from which to choose from for each test item and giving
different candidates in the same test center entirely different versions
of the test.
CompTIA and other certification
organizations have also started to supplement or replace some of the
standard multiple-choice test questions with
adaptive
and performance-based methodologies that are harder to compromise. With
adaptive testing each successive question the user sees depends on
whether or not he answered the previous one correctly. As soon as the
test determines that the taker knows -- or doesn't know -- the content,
the test ends. "It's a more refined manner of judging, but it also
provides security," says Greenwood.
CompTIA is
adding progressively more performance-based testing, which uses
scenario-based questions that ask the user to perform specific actions
in a simulated environment. Such questions are harder to memorize. "At
that point it becomes easier just to study," says Kainrath.
And
that, in a nutshell, is a key part of CompTIA's strategy. "We can't
stop cheating, but we can make sure it takes a lot of time versus just
studying."
Getting caught: A great way to kill a career
Wary
of the damage that rampant cheating can have on an IT certification,
like what some say happened in the 1990s (see sidebar, below), companies
aren't just getting aggressive about catching cheats, they're clamping
down by handing down more severe sanctions.
"We
ban for life anyone who is caught cheating. They are not allowed to take
any Microsoft exam ever again," says Grieve. And Microsoft, at its
discretion, may also strip the candidate of any previously earned
Microsoft IT certifications, she adds.
Devaluing a credential
As
large numbers of people earned the Certified NetWare Engineer
certification in the early 1990s, recalls Dave Meissner, chief operating
officer at Kryterion, "there was concern about the quality of the
professionals being certified. People could pass the CNE exam
successfully purely by studying books," he says, which gave rise to the
term "paper Certified NetWare Engineer." What's more, "there was a
strong belief -- and perception is what matters -- that the test content
was readily available, and the value of that CNE credential was
diminished."
CompTIA is taking a harder line on
cheating as well, "casting a wider net" by using data forensics in its
investigations, says Kainrath. Today if you get caught cheating you
won't get the certification and must wait a period of time, typically a
year, before you can take the exam again. But CompTIA is considering
changing that to a lifetime ban. "This year we'll roll out a harder
policy," he says.
Poyiadgi says that he's seen
cheaters lose their jobs in situations where employers sponsored the
candidates. And if the person was selling test questions and answers, he
or she may be prosecuted by law enforcement as well, he adds.
Kainrath
marvels at the amount of time he says some people spend trying to cheat
their way through IT certification exams. A certification like A+
serves only to validate the user's skills, he says, and if a cheater is
hired or promoted based on false pretenses it hurts the cheater's career
prospects as much as it does CompTIA's reputation. Ultimately, he says,
"It's not doing them any good by faking it."
