spacer
home > ict > spring 2011 > peak performance
PUBLICATIONS
International Clinical Trials

Peak Performance

A fundamental challenge that the pharma industry faces is the lack of industry-adopted clinical trial performance metric definitions. In 2007, the Metrics Champion Consortium (MCC) conducted a survey of sponsors and clinical research organisations (CROs) seeking feedback about demand for and utilisation of performance metrics. Nearly all of the sponsor survey respondents reported that their demand for performance metrics was growing/rapidly growing, but only nine per cent reported having ‘well defined/clearly understood’ metrics in place. Thus, it was not surprising that over half of those who responded did not use metrics to improve productivity or relationships with CROs. Why request metrics if you don’t use them? On the CRO side, more than 80 per cent of CRO survey respondents reported that sponsor demand for performance metrics was ‘growing/rapidly growing’, but none of the CROs reported that sponsor companies requested and reviewed performance metrics to take effective action with them. The survey results clearly showed that while sponsors were requesting performance metrics, they did not understand what was being measured nor how to use the metrics to take effective action.

STANDARDISED CLINICAL TRIAL PERFORMANCE METRICS

To help address the lack of industry-wide standardised performance metrics, the MCC – a not-for-profit organisation – brought together a diverse group of sponsors (biotechnology, pharmaceutical and medical device organisations) and service provider organisations to develop and implement standardised performance metrics with the intent of jointly improving the efficiency and effectiveness of managing and tracking resources needed to successfully run clinical trials. Each sponsor and service provider organisation pays an annual membership fee to support the operations of the MCC. In return, member organisations participate in MCC online meetings, contribute to the collective decisionmaking process of the group in defining and selecting metrics for the standard performance metric set, share ‘lessons learned’ in the collaborative learning environment, as well as have access to the MCC online collaborative workspace. The overarching goals of the MCC are to:
  • Establish defined sets of standard performance metrics that facilitate the management of clinical trials
  • Encourage the use of the standard metrics by sponsors and service providers to identify areas for improvement
  • Facilitate the sharing of metric data among MCC member organisations (through a blinded database)
  • Provide a common basis for constructive discussion between sponsor and service providers that will lead to a common understanding for the need to improve performance

In September 2010, the MCC released the MCC clinical trial performance metrics v 1.0 – a list of 47 performance metric definitions – to its members. These metrics are designed to assess the performance of specific clinical trial tasks, regardless of whether the task is performed by a sponsor or an outsourced partner. Figure 1 shows how the metrics are distributed along a clinical trial process map, with each MCC metric represented by a coloured dot.



As noted on the process map, the MCC metrics are categorised into four types of metrics: cycle time; timelines; quality; and efficiency/cost metrics. Each dot has its own value:

  • The red dot represents cycle time – measuring how long it takes to complete a defined task
  • The green dot represents timeliness – measuring if a task is completed ‘on time’ according to a milestone date or agreed upon time
  • The orange dot represents efficiency – measuring the resources required to complete a task
  • The purple dot represents quality – measuring the number of errors in completing a task, or the completion of quality-related activity
Additionally, more than half of the metrics are identified as ‘leading indicator’ metrics; this means that organisations can use the metrics to identify opportunities to effect change in the current study.

METRICS ARE MORE THAN JUST A FORMULA

The type of information that is provided for each of the MCC clinical trial performance metrics is shown in Figure 2. The detailed metric definitions and descriptions, developed by MCC members, are designed to help organisations gain a better understanding of what is being measured and how to use the results to identify areas in need of improvement. Specifically, the MCC provides member organisations with detailed descriptions of each metric, including:
  • What to measure (definition of terms)
  • How to measure (calculation formula)
  • Why the metric is essential (benefit statement describing what task(s) the metric is measuring and why it is key)
  • How to review the results and when to carry out ‘additional for cause analysis’ (target threshold for conducting additional review)
  • What additional information to review when the target threshold is not met (list of ‘additional for cause analysis’ metrics)
  • Which additional MCC metric results should be reviewed with the metric for a more complete analysis of performance (companion metrics)

Thus, the MCC clinical trial performance metrics provide organisations with standardised performance metric definitions and detailed information about how to use the metrics.  



STUDY START-UP QUALITY SCORING TOOLS


During the development of the clinical trial performance metrics v 1.0, the MCC created a special work group comprised of sponsor and CRO process improvement and quality management specialists. The MCC encourages member organisations to use the metrics to help with process improvement, which determines where to focus improvement efforts, provides information on the current state, and then demonstrates improvements and to maintain the gains made. To ensure that the metrics would meet the needs of process improvement specialists, the MCC asked for the work group’s feedback on the proposed metrics prior to release. Overall, work group participants expressed frustration that the industry focuses on whether clinical trials are being executing in a timely manner but do not measure how well tasks are being completed. Work group participants were pleased that the draft MCC metrics included quality metrics but decided that additional quality assessment metrics were also needed.

Therefore, the work group focused its efforts on developing systematic tools for organisations to assess the quality of study start-up activities. From their collective experience, work group members agreed that starting studies with quality protocols and quality sites are key aspects of running successful clinical trials. Unfortunately, quality is not easy to measure, and the industry does not have a standard means of assessing the quality of these key study startup tasks.

As a result, they developed quality scoring tools for protocol writing and site selection. Both MCC quality scoring tools are self-assessment questionnaires that calculate a quality score based on the responses to key questions. Users complete the quality tool in order to determine how well the protocol development process or site list scores against a set of quality criteria. If the quality score is below expectations, users should consider addressing the areas in which they scored poorly in order to improve quality of the protocol or site list prior to final approval. 

Figure 3 provides an overview of how the protocol development scoring tool works. Once the protocol is ready for assessment, the protocol development team completes the scoring tool. If the quality score is below the target level, the team can decide whether to address areas of concern and re-score the protocol prior to approval – thus providing some quality guidance or risk assessment. Once the protocol is approved, the final quality score should be recorded and organisations can then track how well high scoring protocols correlate to MCC performance metrics such as number of protocol amendments, enrolment and patient retention.



The site selection quality scoring tool follows a similar approach. Once the site list is ready for assessment, the site selection team completes the quality scoring tool. If the site list score is below the target level, the team can discuss how to address areas of concern highlighted by the scoring tool. For example, the team can address concerns by:

  • Adding high additional scoring sites
  • Removing low scoring sites
  • Mitigating the risk of low scoring sites with site training and increased monitoring
In some cases, the low scoring sites may need to be included in the site list for other reasons. To accommodate this, the tool allows you to calculate the composite score without the sites. Thus, the work group designed this tool to provide guidance on the quality of the site selected and highlight sites at risk for poor performance before the site list is approved.

Overall, the focus of both scoring tools should not just be to generate quality scores but to use the scoring tools to assist in rational discussions amongst the team developing the protocol or evaluating and selecting sites. The tools should reward behaviour that supports the development of quality protocols and the selection of quality sites for the trial.

The MCC has launched a member user group programme designed to gather members’ feedback about the tools and collect performance metrics data to determine how well the protocol quality and site selection quality scores correlate to clinical trial performance.

MCC METRICS BLINDED DATABASE

MCC members have expressed a considerable level of interest in having the opportunity to compare their MCC clinical trial performance metric results against other members’ results (in a blinded manner) through a special MCC member database. An executive committee, comprised of MCC members (sponsors and service providers), has developed the database specifications and identified several highly qualified companies who would be suitable parties to be hired to work with the MCC to develop and manage a blinded database for MCC standardised performance metrics data. The key features of the database will include:
  • The ability to compare company results to other members (in a blinded manner)
  • An online metric analytic reporting tool
  • The ability to ‘drill down and roll up’ to different reporting levels
Only MCC member organisations will have the option to participate in the database. The MCC expects to have a proofof- concept database available for review later in 2011.

CONCLUSION

The adoption of well defined, standardised clinical trial performance metrics provides sponsors and CROs with the fundamental tools they need to work together to run efficient, effective clinical trials. Utilising the MCC clinical trial performance metrics, organisations have a better understanding of what is being measured and how to use the metrics to take effective action leading to improved productivity and better sponsor-CRO relationships.


Read full article from PDF >>

Rate this article You must be a member of the site to make a vote.  
Average rating:
0
     

There are no comments in regards to this article.

spacer
Linda Buchin Sullivan is Vice President of Operations at the Metrics Champion Consortium (MCC), a non-profit organisation dedicated to the development and support of performance metrics within the biotechnology and pharmaceutical industry. She is responsible for the day-to-day operations of the MCC, which includes facilitating MCC steering committee and work group meetings, managing the MCC collaborative workspace, and overseeing membership recruitment activities. Prior to the MCC, she was a management consultant for several global consulting companies. Linda received a BSc in Biology and Environmental Science from Trinity College, US. She earned an MBA from The Amos Tuck School of Business Administration at Dartmouth College where she was named a Tuck Scholar. Email: lsullivan@metricschampion.org
spacer
Linda Buchin Sullivan
spacer
spacer
Print this page
Send to a friend
Privacy statement
News and Press Releases

BrainEver and 3P Biopharmaceuticals Collaborate on Development of Recombinant Human Homeoprotein Engrailed-1

Noáin, Spain, November 15, 2022: 3P Biopharmaceuticals (3P), a leading contract development and manufacturing organization (CDMO) specializing in the process development and cGMP manufacturing of biological products and BrainEver a French biotechnology company focusing on discovering and developing First-in-Class therapies for the treatment of neurodegenerative diseases have entered into a long term collaboration to develop hEng1 a recombinant homeoprotein to treat pathologies such as Amyotrophic Lateral Sclerosis (ALS) or Parkinson's disease.
More info >>

White Papers

Bringing more control to temperature-sensitive logistics

UPS Supply Chain Solutions

By 2016, world sales of cold-chain drugs and biologics such as vaccines and blood plasma products will near $240 billion1. The increase has led to a surge in discussion about how to protect these products throughout the transportation logistics cycle. Industry experts are recommending that the term “cold chain” be eliminated from rules, guidance and best-practice documents. “Cold chain” now includes a broader spectrum of temperature ranges, such as controlled room temperature (CRT). This white paper focuses on the initial steps required to establish a temperature-sensitive supply chain.
More info >>

 

 

 

©2000-2011 Samedan Ltd.
Add to favourites

Print this page

Send to a friend
Privacy statement