home > > spring 2013 > future vision

Future Vision

The US Food and Drug Administration is considering decentralising the interpretation of images in clinical trials measuring progression. As this proposal moves ahead, what would it mean for sponsors?

Historically, the US Food and Drug Administration (FDA) has recommended that when radiological images are used to determine a key efficacy or safety endpoint in a clinical trial, they should be reviewed and evaluated centrally. Last summer, the FDA’s Oncologic Drugs Advisory Committee (ODAC) discussed a potential alternative approach for oncology trials in which progression-free survival is a critical endpoint. The proposed change would allow investigators to make endpoint assessments themselves, with only a sampling of images reviewed centrally as a test of reliability. Trial sponsors considering this approach should evaluate the decision with great caution. 

Value of Central Reviews

The rationale for the use of central review should first be placed in historical and scientific context. There are sound reasons why central review has always been the standard. Though the FDA, in its guidance published in 2007, has not mandated central image review, it has noted various possible difficulties with local reads. These issues include:

It is difficult to ensure that a local reader remains blinded to the patient’s study arm; in the clinical care setting, oncologists and radiologists often confer and review scans together. Distinctive side-effect profiles of many oncology drugs may result in functional unblinding of the clinician, who can then influence radiologists during a review. 

The interpretation of radiological images is open to natural, random variability. Any two readers will disagree on the date of progression about 30-50 per cent of the time, depending on tumour type, rate of disease progression and other factors. The use of multiple readers on each case increases the accuracy and reduces the variability of interpretations. Central facilities decrease the likelihood that the date of progression will be determined incorrectly through a ‘2 + 1’ read design. Two primary readers assess progression and a third reader adjudicates any disagreements between them. Additional variability arises from the varying experience and training of the readers across sites.

Clinical Trial Training
Formal tumour response criteria, such as those used in clinical trials, are not part of a radiologist’s clinical training or daily practice, so reader training is essential. A recent survey of cancer sites with approximately 1,100 responders revealed that, even at academic centres, images are read by a radiologist dedicated to the study only 41 per cent of the time. Most of the time, reads are performed by the radiologist on call, a non-radiologist investigator, another physician, or even other study staff such as research nurses and coordinators. A significant proportion of sites reported that readers were trained in trial criteria by selflearning, or had no training provided at all. Clinicians are also not routinely trained in good clinical practice and other regulatory compliance guidelines. 

Data Quality
In the 1990s there were several prominent instances in which local interpretations of images created serious problems with trial data quality. The FDA performs quality assurance audits on central laboratory facilities that perform image interpretation. During such audits, the facilities are asked to explain their interpretation process and to defend their results for selected cases.

Proposed Approach

At the ODAC meeting last summer, statisticians from the National Cancer Institute (NCI) and the pharmaceutical industry presented study results suggesting that the factors described above may not affect the quality of endpoint determination to the degree imagined. Across 27 solid tumour trials, the aggregate results (the hazard ratios showing treatment effects) were highly concordant. It is also important to mention that there were methodological concerns about the studies, including a large number of missing scans (up to 30 per cent in some cases). Two different statistical methods were discussed for detecting bias in local reads – each involving a comparison of the aggregate results from a sample of central reads to the results from local reads.

Therefore, the ODAC recommended to the FDA that sponsors should be able to “rely on investigator assessments, augmented by audits (central reads of a sample of the assessments) designed to detect bias”. In the proposed approach, local evaluators would determine when progression has occurred and then send their scans to a central facility. The central facility would conduct a blinded review of a sample of the scans and compare the results to those of the local investigators. This statistical audit (not to be confused with a typical regulatory audit of how case decisions are made) would not evaluate agreement on individual cases, but rather test for statistical evidence of bias. If the audit confirmed the reliability of the local evaluation, the results would be accepted. If there were any indications of concern, the process would revert to the prior standard, requiring a complete central review. In simulations conducted by Dr Lori Dodd at the NCI, a complete central review was required 16 per cent of the time.

Cost Analysis

Since the goal of the change would be to cut trial costs, it is important to examine the cost implications carefully and develop a realistic view of the potential savings.

The cost of performing central reads today amounts to only a small fraction – one to three per cent – of the total cost of conducting a Phase 3 trial. So, for a trial with a budget of $100 million, the central read costs are one to three million dollars. In the best scenario, where local assessments do not incur any expense and a complete central review does not result, a portion – but certainly not all – of this expense could be saved through the proposed process.

Continued Costs

Under the new process, sponsors would still incur image-related costs for:

● Prospective image collection and quality control. Since there is a possibility that a complete central review would be needed on any given trial (when images are collected retrospectively, typically about 30 per cent of image data is unavailable)

● Developing auditable documents, such as the imaging charter, to describe the process of data collection and the independent review

● Training sites to obtain and submit protocol-compliant images

● Training readers at both central labs and local sites

● Project management

● Central review of the sample images (the simulated audit methods presented at the ODAC meeting used sample sizes ranging from 25-45 per cent of the total number of cases in each trial)

● A complete central review of all images over and above the local reads, should the audit indicate bias in the local reads (note that 16 per cent of the trials analysed for the ODAC meeting would have required a subsequent, complete central review)

As an illustration, cost savings that would be realised under the proposed process for nine recently conducted Phase 3 solid tumour trials, each with an average of 700 subjects, has been calculated. Assuming that 30 per cent of the total number of images would form the sample to be subjected to a central review, and that all other activities would be unchanged from what is required today, the results point to an estimated 22 per cent saving on image-related costs, if no complete central reads were needed. If, as in the simulations presented at the ODAC meeting, there is a 16 per cent chance of a trial requiring a complete central read following the local read, the average savings per trial drops to 18 per cent of image-related costs. For a trial costing $100 million, the expected savings would therefore be around $520,000, or 0.5 per cent of the entire cost of the trial.

Added Costs

However, adopting the new process would introduce other costs that would, to some degree, offset the above estimated savings. These costs, which have yet to be quantified, would arise from the need to:

Increase Trial Size
An increase in the variability of endpoint measurements may mean that trials will need to involve more subjects. Previous sample sizes are primarily based on the variability resulting from a multiple-reader protocol, and a larger variability should be expected.

Reimburse Local Readers
While contracts are currently made with investigators who are responsible for reimbursing local radiologists as needed, sponsors would need to develop contracts directly with sites’ imaging departments.

Comply with FDA Standards
Individual investigator sites may need help with preparing for training, procedures and documentation in order to comply with FDA regulations.

Develop Additional Statistical Methodologies
Building the sampling methodology into the trial design and developing the technique for comparing the central read sample to local reads will require the services of PhD-level statisticians. Any interim statistical audits not done by the sponsor will need to be contracted. 

Cope with Delays
Should the statistical audit reveal a bias in the local reads, all scans will need to be re-read centrally. This duplication of effort will, of course, add to the reading costs, and most certainly delay the close of the trial, which could have a significant financial impact.

So, while the costs associated with these aspects of the new methodology are not yet known, it is quite reasonable to assume that they will dramatically reduce the possible savings and could even lead to a net increase in image related costs.

Other Unknowns

At the time of print, the FDA had not published an opinion on the ODAC proposal, and it may be some time before it does. The agency does not change its established guidance without thorough study and input from various stakeholders and advisors. It does, however, seem likely that these recommendations will have some influence on future FDA guidance. Even so, the new process would not be appropriate in all situations; the ODAC has specified that the new methodology would probably not be suitable for small trials, when the expected effect size is moderate, or when the type of tumour being studied is particularly difficult to assess.

What is more, the ODAC recommendation was a broad procedural outline; many details remain unsettled and many questions unanswered. For example:

● What percentage of scans needs to be sampled?

● How should the sample be drawn? For example, a completely random sample might over-represent some sites and miss smaller sites completely

● At what point(s) in the trial should the sample be studied centrally?

● What statistical methods should be used to compare the results from the central interpretation to those from the local interpretation?

● What would be the acceptable level of disagreement between the central and local interpretations?

● How would the situation be handled, should the level of disagreement exceed the established limit?

In all likelihood, sponsor companies will need to resolve these issues by putting forth their own proposals to the FDA on individual trials.

Strategies for Minimising Risk

In the event that local image assessments are used as the basis for a drug approval, it should be expected that the FDA will conduct regulatory audits at investigator sites, similar to those it now conducts in central facilities following successful trials. In such an audit, local sites would be expected to document every step of the process involved in determining an endpoint for selected patients. Who performed the read? How was the reader trained? Which images were examined? Which lesions on an image were measured? What measurements were taken? How were the measurements combined to form an assessment of response? How was the endpoint derived?

Given sponsors’ vested interest in ensuring that investigators pass such audits, they should consider how they might help local sites prepare for regulators’ scrutiny. Options for sponsor or central facility involvement range from a relatively relaxed approach to a very rigorous one, as outlined below:

Low-Touch Approach
Sponsors could simply allow investigators to follow their own processes and procedures for reading images and documenting their work, perhaps offering some guidance and oversight. Sites would submit their endpoint determinations and all scans to a central lab. The lab would then perform the sample read and make the statistical comparison between the results of the central and local reads. No special preparation would be taken to ensure that site reads would stand up to a regulatory audit.

Medium-Touch Approach
Sponsors could decrease the likelihood of a significant discrepancy between the results of the local read and of the centrally-read sample images by increasing training, control and documentation requirements for sites. The central facility could train local readers, so all readers would have a common understanding of the criteria and follow systematic procedures for performing, tracking and managing their reads. The central facility could also periodically conduct spot checks at local sites to ensure that they were applying their training uniformly. In this model, local readers could also provide documentation on how they made their endpoint determination, which could be collected by the central facility. For example, screenshots of lesion measurements and copies of forms used to derive responses could be collected, in case they were needed for later audit purposes.

High-Touch Approach
In a more sophisticated arrangement, the central facility could deploy an electronic work environment that local readers would use to enter their evaluations. Such a system could be accessible via a web portal, and allow review of images and entry of interpretations through a robust process in which every local reader’s analysis and final determination would be captured in a controlled environment that would provide a full audit trail.


It remains to be seen how the FDA will proceed. A local read method with statistical auditing is a departure from a great deal of FDA experience and history. However, there was support for this approach during recent discussions, so it may, in the future, be cautiously permitted for a certain population of trials. The prospect, while intriguing, may not yield the windfall in cost savings that sponsors might expect, and could add to trial complexity. The European Medicines Agency, in its own guidance issued in December 2012, discussed the possibility of this approach, and stated that: “Regulatory guidance on the appropriateness of any such approaches should be sought on a case-by-case basis before implementation to discuss, in particular, ensuring integrity of the study, how the sample will be generated, and the statistics and metrics to be used for deciding whether or not an important directional discordance can be excluded.” Sponsors should, therefore, proceed with caution, exploring the proposed methodology with their eyes wide open, consulting with statisticians and operations experts on the best way to comply with regulators and meet their study goals.

Read full article from PDF >>

Rate this article You must be a member of the site to make a vote.  
Average rating:

There are no comments in regards to this article.

 You must be a member of the site to make a comment.
Gregory Goldmacher is currently Director of Medical and Scientific Affairs and Head of Oncology Imaging at ICON. Gregory is a radiologist and provides medical, scientific and regulatory leadership for trials in every phase of clinical development, as well as training physicians and study staff worldwide in clinical trial imaging methods.

David Raunig has 15 years’ experience as a research statistician in the pharmaceutical industry, from discovery statistical analysis through to Phase 3 clinical trials. As ICON Medical Imaging’s Vice President of Informatics, David brings a great deal of knowledge and expertise to the study, design and analysis of imaging biomarkers and reader performance for all modalities.

James Conklin is the Senior Vice President for Medical and Scientific Affairs at ICON. James has over 30 years of experience in the use of medical imaging in clinical trials. He is a Johns Hopkins-trained research physician who is board-certified in internal medicine and nuclear medicine.
Gregory Goldmacher
David Raunig
James Conklin
Print this page
Send to a friend
Privacy statement
News and Press Releases

WCCT Global (an Altasciences Company) Clinic Expansion

WCCT Global, an Altasciences company, announced the expansion of their clinical pharmacology unit and laboratory earlier this year.
More info >>

White Papers

Cold-chain Bioprocessing Readiness: Mitigating Risk and Protecting Pharmaceutical Products


The pharmaceutical industry is migrating rapidly toward a world where drugs, vaccines, and specialized therapies are available on demand to patients anywhere around the globe. Whether accelerating clinical trials to bring new drugs to market or devel oping biosimilars to give more patients access to established commercial drugs, small and midsize production facilities stand to benefit from growing demand. To do so, however, they must be prepared to handle a broader mix of products and increase throughput without sacrificing safety or reliability, while at the same time keeping life-saving medica tions affordable. Such challenges become more difficult when working with products that must be stored and shipped in a frozen state.
More info >>




©2000-2011 Samedan Ltd.
Add to favourites

Print this page

Send to a friend
Privacy statement