[ RadSafe ] Peer-Review / Checking Bibliographic Citations

[Dan W McCarn]

Dan W. McCarn, Geologist; 3118 Pebble Lake Drive; Sugar Land, TX 77479; USA

Dear Steven & Jeff:

In the larger setting, review can be for several objectives:

1) Report prepared for licensing of a nuclear facility *
2) Paper submitted to a refereed journal (See Jeff's comments)
3) Paper submitted in a proceedings *
4) Affidavits submitted for a regulatory hearing *
5) Depositions / Affidavits / courtroom testimony as expert witness (later)

Each of these categories have different review requirements, so "It
Depends".  

For  a document as important as the Yucca Mountain Site Characterization
Report, every reference was obtained, reviewed, and either validated by
professional staff or the reference was kicked back to the original authors
for more specificity and change.  This was all done as an iterative process
including the formal peer review on each section, and the peers were
provided access to the complete text of all references. Once the initial
review was received by the authors from the peers, the reports were revised
and usually changed for the better. Sometimes this process iterated a couple
of times. For unresolved issues, the peers and authors met in a closed room
with a mediator and would proceed to work-out their differences.  Usually
these meetings were collegial.

For refereed papers: I am in agreement with Jeff: << I, always, skip those
that are not relevant to the major point of the paper. For example, in the
introduction of a paper you usually get the litany of these references are
similar to the work described. I do read those that are germane to
controversial points in a paper and often suggest other papers that bolster
or contradict that authors comments.>>

Citing an entire 600 page textbook or volume makes no sense whatsoever.  Use
of such generalized citations runs a red flag up the pole for me.  When I do
check citations for a paper, I will tend to target the most vulnerable
assumptions and statements, or at least try to frame the range of validity.
I will then attempt to prove them invalid. e.g. “Calves are never born with
a cleft lips”.  If I can easily invalidate such a statement through multiple
references, I send the paper back with those references and a negative
review until that portion is modified or rectified.  But this kind of black
and white, simplistic review usually occurs in papers that are not well
considered.

Proceedings: Papers submitted to a proceedings, have the least review.
Sometimes they are simply "accepted" or "rejected" by the meeting organizers
or the scientific secretary and sometimes review is limited to spelling and
grammar. Sometimes "peer review" is not warranted, e.g. a country, company,
organization or regulator reports of their activities in an area.
Occasionally, discussions of presented papers are also published and these
are occasionally very helpful.

Other times papers submitted in a proceedings of a meeting and are written
for an audience other than purely scientific e.g. business, advertizing,
politics, etc.  These types of papers may range from excellent to terrible.
Papers in poorly managed proceedings occasionally are invalid through their
own over-reaching attempt to meet the politically driven conference
objectives or simply "seem" scientific in very sketchy terms. Clearly papers
like Fathallah (2007) or the Basrah paper were never peer-reviewed or
provably invalid assertions would have been removed or significantly
modified. Floating assertions with no support are sadly common.
Unfortunately, so much time frequently goes into invalidating these papers
if they becomes a favorite of the public advocates of a specific political
position.

Affidavits for regulatory hearings are usually reviewed by professionals in
that field. 
When I submitted affidavits to the NRC, I also provided full text for all
references cited for the Hearing Officer including sometimes an entire
published volume (if there were multiple references to papers in a volume).
Almost to a fault, the opposition would reference portions their own
previous affidavits that a previous hearing officer had rejected as invalid.
They have to be re-challenged again every time!

For courtroom testimony as an expert witness, so much depends on the ability
of the court, the judge and jury to understand the scientific argument. This
last type can be the most challenging and difficult but is often settled out
of court during the discovery phase and when formal depositions are
presented.

Dan ii


From: Steven Dapra [mailto:sjd at swcp.com] 
Sent: Friday, May 23, 2008 10:52 PM
To: Dan W McCarn; radsafe at radlab.nl
Subject: RE: [ RadSafe ] uranium smoke is a teratogen

At 12:54 AM 5/23/08 -0500, Dan W McCarn wrote:

[edit]


One last thing: not all scientific papers in this day and age are
peer-reviewed.  In fact, I'd say that only a small fraction are effectively
reviewed especially in the reports that you mentioned.  When I worked on the
Yucca Mountain Site Characterization report, an enormous effort was taken to
validate every citation in preparation for a formal peer review. It is not
sufficient to just "pile on" references, they must be cogent and meaningful
and very specific. There was a time when I had more references checked-out
of the Yucca Mountain library than all other workers put together.  I even
had my own reading room for organizing the peer review.

[edit]

        This is a very good point, and one I have pondered at times when
reading papers in Health Physics and other refereed journals.  Do the
authors of these papers take the time to read all the literature they cite? 
Can they take the time?  Do they even read the paragraph wherein the
sentence appears that they have quoted?  We often see papers with 30 or 40
or 50 (or even more) references.  No one can read all that stuff before
writing a paper --- can he?  And to imagine the reviewers reading all that
material to ascertain that it has been used properly?  Not to detract from
the valiant efforts of any reviewer, I do not see how it would be possible
for anyone to read all that stuff.  I don't think it would be possible to
check all the references and ensure that they are correct --- meaning
correct volume, date, and page numbers.

        I emphatically am not denigrating anyone's reviews.  I am merely
noting the enormous amount of work it would be to thoroughly review a paper.

Steven Dapra