Ethicist questions insurance rate data
TAMPA, Fla. – Jan. 15, 2007 – A new computer model that has contributed to large spikes in property insurance rates in coastal areas should have had full review by outside experts before it was released, the ethics chief of the world’s largest general scientific society said.
Research Management Solutions of Newark, Calif., introduced the product to the market long before the scientific peer review process was finished.
“It’s ridiculous from a scientific point of view. It just doesn’t wash well in the context of the way science is conducted,” said Mark S. Frankel, director of the Scientific Freedom, Responsibility & Law Program at the American Association for the Advancement of Science in Washington.
In March, RMS announced a major change to the leading computer modeling program for the insurance industry and made the update available to its clients.
The program predicts a 40 percent increase in losses from hurricanes in Florida, the Gulf Coast and the Southeast. Although not yet approved by the state for use by Florida insurers, it is being used by reinsurance companies that provide financial backing to insurance companies, indirectly increasing rates. The cost of reinsurance makes up 30 percent to 60 percent of premiums, according to the state.
RMS has said development of the new short-term model, which focuses on weather from 2006 to 2010 instead of depending on 100 years of historical data, was well documented.
The company said Thursday that the results are in the process of being published in Tellus, a journal put out by the Swedish Geophysical Society.
That’s the wrong order for several reasons, other experts said.
“In the future doesn’t count” in terms of peer review publication, said Sara Rockwell, director of the Office of Scientific Affairs at the Yale University School of Medicine.
Charles Watson, an engineer who specializes in numerical hazard models, said RMS acted irresponsibly.
“You publish; then you apply it,” Watson said. “Especially for something with trillions of dollars in property value, and people’s lives and livelihood are literally at stake in these decisions. It is irresponsible to implement before peer review. There are tremendous policy implications.”
The peer review process is used throughout the scientific community. Researchers submit their theories and findings to independent journals; then editors assign other experts to review the material and raise questions.
After that review process the material may be published, but even then other scientists have the opportunity to raise questions and publish rebuttals.
Not using peer review from the start is as if someone invented a drug and used the product on patients before an independent review showed it to be safe.
Research submitted for publication
On Thursday, Robert Muir-Wood, RMS’ chief research officer, said in a written statement that RMS scientists started presenting the new model at a climate conference in late March, soon after it announced the change. The company submitted the research for publication in Tellus on Aug. 31, and final publication is expected this year.
Muir-Wood also mentioned the “expert elicitation” process RMS conducted in October 2005 – when the company paid the expenses for four scientists to meet in Bermuda and discuss the issue. The company later mentioned the scientists in news releases and included pictures of them in a slideshow on the new model.
Last week, two of those scientists told the Tribune they didn’t agree with some of the statements RMS has made about the model and noted that they only had a chance to review a portion of the data in question.
“The expert elicitation is one of the most effective ways of gaining a peer review,” Muir-Wood said, and the company conducted a second elicitation in the fall. Frankel of AAAS said that’s not real peer review.
“First of all, he’s depending on the views of four scientists,” and that’s too small a number, especially because they accepted something of value from RMS, Frankel said.
“This is a really touchy issue in science generally,” Frankel said, and the scientists may not have been clear – or asked all the right questions – about what was expected of them in the elicitation process.
Frankel was particularly surprised about the participation of Thomas R. Knutson, a research meteorologist with the National Oceanic and Atmospheric Administration in Princeton, N.J.
The participation of a government climate scientist in a process that has a direct effect on the price of insurance is particularly sensitive, Frankel said.
Knutson told the Tribune last week that the five-year timeline didn’t come from the experts.
“I think that question was driven more by the needs of the insurance industry as opposed to the science,” said Knutson, who also questioned the extent of some of the RMS projections about hurricane landfall.
RMS says all the scientists signed off on the process.
Frankel made clear that although he wasn’t in a position to criticize the RMS model, the company’s failure to follow the peer review process has major public policy implications, he said.
The fact that the RMS model is such a dramatic shift from previous versions was even greater reason for a full, advance independent review, Frankel said.
“It’s becoming something that’s much bigger. This could really open up an interesting can of worms for this whole area of developing models,” he said. “If you were relying on that for decision-making. … I mean, my goodness!”
For Florida consumers, there’s a particular irony in the RMS move to a five-year model, said Steve Burgess, Florida’s insurance consumer advocate.
“We used to use a seven-year average before Hurricane Andrew” hit South Florida in 1992, Burgess said, but the insurance industry complained that was too short and demanded the move to long-term forecasting.
Hurricane loss models deemed unreliable
Watson, who conducts research with University of Central Florida statistician Mark Johnson, said their work indicates that hurricane loss models are unreliable under the best of circumstances.
Watson said he and Johnson think more reliable figures are generated by looking at hundreds of models in the aggregate. Though they vary wildly, they seem to generate consistent means. Florida also has a public hurricane model to check rate filings for residential homes.
Some Florida politicians have just started to question the potential effect of the new RMS model.
On Thursday, Sen. Steven Geller, the Democratic minority leader from Hallandale Beach, said he didn’t think the Senate would call on RMS officials to testify under oath during a special session on insurance next week.
But when Geller was told of RMS’ shift to a short-term model and its prediction of a 40 percent rise in insurance losses, he grew concerned.
“The problem is that historical data is historical, while other things are guesses,” Geller said. “I would certainly have concerns.”
Another major modeling company has similar concerns about short-term models and the RMS prediction that more storms are likely to hit specific regions of the country during the next five years.
“There is no consensus in the scientific community that there is a clear link between activity in the Atlantic basin and landfall activity, and where storms are likely to make landfall,” said Jayanta Guin, vice president of research and modeling at AIR Worldwide in Boston.
Still, AIR reacted to the RMS move and has released its own short-term model, though it suggests it not be used to replace the original long-term model.
Copyright © 2007, Tampa Tribune, Fla., Kevin Begos and Michael Fechter. Distributed by McClatchy-Tribune Business News.