Big Glitch at Nuclear Plant Shows Perils of Y2K Tests

["Sandy Perle" <sandyfl@earthlink.net>]

In today's Washington Post

Big Glitch at Nuclear Plant Shows Perils of Y2K Tests  

Sunday, March 7, 1999; Page A03  

PEACH BOTTOM, Pa.—In a sterile room filled with rows of hulking 
1960s-era mainframes, complete with panels of blinking red and 
green lights, a half-dozen technicians at the massive nuclear power 
plant here set out early last month to test whether one of the 
facility's critical computer units would understand the year 2000.  

It was supposed to be a prosaic affair. The unit in question had 
been pored over by programmers, it had been analyzed for a week 
in a simulator, and it was being hooked up to a backup version of 
the facility's central operations monitoring system.  

But when the computer's clock was turned ahead to Jan. 1, 2000, 
something went drastically awry at the Peach Bottom Atomic 
Power Station.  

In what experts say was one of the more serious computer glitches 
in recent memory at a nuclear plant, the facility's primary and 
backup operations monitoring systems--which provide control-room 
technicians with vital data about temperature, pressure and water 
levels in the reactor's core--crashed. Every computer screen in the 
plant's control room blacked out and froze, forcing technicians to 
rely on antiquated gauges.  

Plant managers say the incident posed no risk to the public, but 
they nevertheless began planning to shut down the facility, which 
supplies electricity to the Philadelphia area. They eventually 
scotched those efforts after the computer specialists determined 
the source of the problem--a technician had improperly set the test 
clock--and restored the systems seven hours later.  

Although the cause was human error, technology specialists say 
the glitch here illustrates an unanticipated peril of the Year 2000 
problem: As computer systems that have been repaired are now 
being tested in live conditions, inadvertent mistakes and 
undiscovered bugs can bring the machines--and the organizations 
that rely on them--to a grinding halt.  

"When you perform tests, you inevitably create some errors," said 
John C. Ballock, a Year 2000 manager for Computer Sciences 
Corp. With intensive Year 2000 testing taking place at almost every 
business and government agency around the world, such glitches 
are "something that we're definitely going to see more of," he said.  

In Missouri, for instance, about 50,000 residential customers of the 
Metropolitan St. Louis Sewer District received incorrect bills last 
month--each was charged for consuming exactly 800 cubic feet of 
water in January--after a programmer failed to remove Year 2000 
test data from a billing system.  

In Texas, 2,013 customers of Bank One Texas received erroneous 
notices in December saying they had bounced checks after an 
employee accidentally mailed out overdraft notices that had been 
printed for a date-related test.  

And in Illinois, the village of Oswego got a monthly electric bill late 
last year for $7 million--about $6,989,000 more than the town 
normally is charged each month--because of software "bugs" in a 
new computer system purchased by Commonwealth Edison Co. to 
address the Year 2000 issue.  

The Year 2000 problem--dubbed Y2K--stems from the fact that 
millions of electronic devices, from mainframe computers that 
process payroll checks to heart monitors in hospital intensive-care 
units, were programmed to recognize only the last two digits of a 
year and to assume that the first two would be 1 and 9. When Jan. 
1, 2000, arrives, unrepaired machines will understand the year "00" 
not as 2000 but 1900, potentially causing them to shut down or 
stop working properly.  

Despite the glitches that have been cropping up, technology 
analysts say testing is a critical part of the repair effort. It makes 
sense, they say, to discover bugs and to deal with any instances 
of human error now instead of later this fall or on Jan. 1.  

"The fact that people are having problems now is a good thing 
because it at least shows they're testing," said James Woodward, 
a senior vice president at Cap Gemini America, an information 
technology firm that provides date-related repair services.  

To that end, 400 U.S. banks, brokerage firms and stock exchanges 
embarked yesterday on the first stage of a seven-week program to 
test their ability to trade stocks and other securities in the new 
millennium. The test is being closely watched by technology 
analysts because the securities industry has long been viewed as 
a leader in addressing the Y2K issue and any problem could 
presage difficulties in other sectors of the economy.  

On Merrill Lynch & Co.'s football field-sized trading floor in New 
York, a dozen traders spent the morning pretending it was Dec. 29 
and conducting about 500 simulated transactions.  

For trader Jason Horowitz, 29, the drill involved buying 1,000 
shares of Costco Wholesale Corp. at $71.75 apiece from fellow 
securities firm Lehman Brothers. Working from an elaborate script 
that specified just how many shares of what to buy from whom at 
which price, Horowitz banged away at his computer terminal, 
quickly entering abbreviations for the required information.  

A few seconds later, the magic words appeared in blue text on his 
screen: "Transaction Confirmed."  

"This feels just like any other day, except that I'd probably be 
wearing a tie," said a casually dressed Horowitz, as he entered 
orders to sell 1,000 shares of 3Com Corp. and buy 1,100 shares of 
Intel Corp. All told, he executed 13 trades and received similar 
confirmation notices. "Everything seems to be going fine," he said.  

Merrill executives and industry officials generally concurred with 
that assessment, although they said they would not have detailed 
results until an auditing firm analyzes the trading data next month.  

"There probably will be some problems that come out of these 
tests for some of the smaller firms that haven't spent as much time 
fixing their systems," said Arthur L. Thomas, a Merrill senior vice 
president who is overseeing the testing efforts. "But we're 
expecting them to be minor and isolated."  

The securities firms decided to begin their tests by simulating Dec. 
29 because transactions executed that day will be settled on Jan. 
3, the first day of trading in the new year. The tests will continue on 
Saturdays until April 24. All told, the industry plans to conduct 
about 170,000 mock trades.  

Other industry groups have conducted--or are planning to perform--
similar drills. The Telco Year 2000 Forum, an organization 
representing the nation's largest local telephone companies, 
released a report on Wednesday detailing the results of cross-
company tests conducted in the past six months. Of 1,914 test 
cases, only six resulted in Y2K "anomalies," the group reported.  

"We remain confident that dial tone will be available on January 
1st," said Gene Chiappetta, the forum's chairman.  

The North America Electric Reliability Council plans to conduct an 
industry-wide test next month that will simulate a partial loss of 
computer systems and another in September that will simulate the 
rollover from Dec. 31 to Jan. 1.  

Some industry consultants have expressed worries that conducting 
broad tests late in the year will give companies little time to fix the 
problems they uncover, but they say they are encouraged by the 
pace of individual efforts in places like Peach Bottom.  

"It's good that these problems are happening now," said David 
Lochbaum, a nuclear safety engineer with the Union of Concerned 
Scientists. "You don't want this to occur on Dec. 31."  

At Peach Bottom, the problem began just after lunch on Feb. 8, 
when a group of technicians tested a computer called the 
"Rodworth Minimizer." The unit, which operates when the reactor is 
at low power, analyzes the position of "control rods" in the core 
and tells engineers which ones can--and cannot--be removed to 
balance power distribution.  

To simulate Jan. 1, the technicians had intended to connect the 
Rodworth to another computer that would serve as a clock. But 
instead of connecting the unit to the external clock, a programmer 
inadvertently reset the date on the backup and primary operations 
monitoring systems, which are not yet Y2K compliant, said Joseph 
Clepp, an information systems manager at Peco Energy Co., the 
Philadelphia-based utility that runs Peach Bottom.  

As soon as the date was reset, the screens in the control room 
went blank.  

"There was no panic, but there was a lot of concern," said Jim 
Kovalchick, a shift supervisor who was in the control room during 
the outage. "We don't like unusual things, and here we had the 
high sensitivity of a system going down."  

After a discussion in which "we tried our best to be thoughtful 
about the situation we were in," Clepp's team decided to restore 
the operations system in a piecemeal fashion, testing each part as 
they went along. "We didn't want to bring up the system only to 
have it crash again," he said.  

The whole task took seven hours.  

The Nuclear Regulatory Commission, which said the Peach 
Bottom incident was the first Y2K problem at a nuclear plant, 
appeared sympathetic to Clepp's travails. "With computer software, 
it's hard to anticipate all the difficulties you can run into," said 
Jared S. Wermiel, chief of the NRC's reactor systems branch. "And 
too often, it's what you haven't thought of that comes back to bite 
you."

------------------------
Sandy Perle
E-Mail: sandyfl@earthlink.net
Personal Website: http://www.geocities.com/capecanaveral/1205

************************************************************************
The RADSAFE Frequently Asked Questions list, archives and subscription
information can be accessed at http://www.ehs.uiuc.edu/~rad/radsafe.html