K-Blog

Converging Data Challenges: How to Help the FDA

2010-06-10T11:32:00.004-05:00

The FDA and DIA collaborated on a meeting in March on the data challenges faced by the FDA, and the role that, among other things, industry and standards play in making their actions more efficient. Kit wrote an article on the meeting and its implications for industry and standards, which was published in ClinPage. You can find it here:

http://www.clinpage.com/article/how_to_help_fda/C9

EDC Help Text or No EDC Help Text: That is the Question

2010-04-21T15:30:00.007-05:00

Should EDC applications should contain completion instructions and/or help texts associated with individual fields or eCRFs?

Some believe that if the questions are worded clearly, the cursor control is logical and helpful, the layout is clear and uncluttered, and appropriate data capture structures are used (e.g., radio buttons, dropdown lists), then there should be no need for additional instructions or help as the form should stand on its own. To address differences between sponsors in layout, cursor control and question wording, a study conventions document can be prepared that contains general information regarding entering data and the like.

I believe that this is only part of the picture. Without question, EDC prompts should be very clear, use appropriate field structure, and so forth. These characteristics are tightly linked to defining and capturing high quality data. It is also not necessary to create help texts that only restate the prompts, e.g., a help text of “Please enter the subject’s weight” on a field with the prompt “Weight.” Finally, a study conventions document can provide valuable information about navigation, handling individual fields with missing data (e.g., put a note in the pop-up query window), and the like.

On the other hand, such a document does not educate the sites with respect to many of the sponsor’s other expectations for data quality and consistency, which can be thought of as the scientific or business rules, and this can be quite a thorny issue. For example, the study conventions document does not provide specific information on the edit check ranges used on the fields, nor why those ranges were chosen. It doesn’t usually alert the site to data relationships the sponsor expects to see, such as AEs corresponding to each new concomitant therapy started during the study. In fact, virtually everything that the sponsor expects check and that could result in a query should be communicated to the site in some way. Otherwise, how is the site supposed to know how the data should be captured and recorded? It is like requiring them to take a test on material that they have never been taught!

Some of the information mentioned above could be drawn from the protocol, but many protocols do not go to this level of detail, and given that study coordinators often run many studies with different sets of rules, it would seem safer to add the rules to the tool they use to prompt them for those data. The information is also in the Data Management Plan, but most sites do not see that. Given the amount of disagreement there is on chat forums such as LinkedIn, it is clear that there is no consensus on many of these practices, so how can we expect the sites to “just know” what to do? Here are some more examples of scientific or business rules that I have seen vary either on discussion forums or between clients:

When in a study to start capturing AEs and SAEs?
How far from the protocol-defined visit day can a visit occur and still be “compliant”?
Over how many days can a visit occur and still be “compliant”? E.g., bloods one day, physical exam the next, ECG the following.
How should open AEs and concomitant therapies be handled if the subject dies during the study (i.e., should they be closed out with date of death as the end date, or left open)?
What conditions should be recorded on the medical history screen? I.e., starting at what time point, or “clinically significant” ones only (though that definition is not always clear)?
Should symptoms or diagnoses be recorded on AE eCRFs?

If these questions and many more like them are not defined for the sites in a way that is clear, accessible and user-friendly, then we will generate ever more queries and worse, the data will be less comparable and uniform (i.e., lower quality), and we may not even know it.

New FDA Guidance Requires More Data for Every Trial

2010-01-21T22:03:00.003-05:00

In July 2009, the FDA issued a new final guidance defining clinical trial design, data and analysis requirements for drug-induced liver injury, or DILI, cases in all drug and biologic agent clinical trials.

DILI occurs in of 1 in 10,000 patients or less, and currently there is no way to predict what drugs may cause the condition, nor which subjects will be affected.

The guidance lists some specific data points that must be collected for subjects who experience DILI that go well beyond the data that is usually collected for most subjects in most trials.

This will require some considerable rethinking of how we capture and store some types of data. If you’d like to learn more, please see the lead article in the January 2010 edition of K-News, available on the News page at www.kestrelconsultants.com.

Complexities in Reporting SAEs: What Do the Regs Really Say?

2009-12-07T12:21:00.007-05:00

“If a patient is randomized to the study but never receives study drug, must site staff report serious adverse events (SAEs) for this patient?”

This excellent question was recently asked in the CDM group on LinkedIn. The responses ranged from assertions that regulations require that either serious or all AEs must be reported starting at informed consent (IC), to suggestions that, in the absence of any internal company guidance, they should be captured as they can be eliminated from the analysis based on the “definitely not related” assessment.

I think the answer is probably “it depends”! As with most aspects of clinical research, deciding what to collect and to report requires understanding the risks associated with the different choices. That requires understanding, among other things, the context of the question, where and when the research occurred, and who intends to use the data and for what. Below you will find my thoughts on some of these factors.

(Incidentally, if anyone can point me to a place in a regulation where it defines “on study” as beginning with signing IC, and/or where it clearly states that SAEs must be captured beginning at IC, I’d be grateful. I was unable to locate any such statement, but my sources are primarily US regulation and guidances, ICH guidances, ISO standards, and some European regulations and guidances.)

The Question

The original question was whether SAEs occurring after randomization but before treatment must be reported. Because of the range of responses in the Group, I have also expanded the question to “Must all AEs and/or SAEs occurring on study be reported?”

There are two elements to consider here.

The first is the meaning of “reported.” Does it mean that AE/SAEs must be reported on a CRF? Or does it mean that AEs/SAEs must be reported to regulatory authorities under the expedited reporting rules? Or does it mean that AEs/SAEs must be reported in the study report? Each of these is a valid question, and the answers depend upon a number of factors, some of which are discussed below.
The second is the meaning of “on study.” Is it when IC is signed, or at randomization, or the beginning of baseline, or something else? One could use the term “enrolled” instead, but it turns out that neither term is clearly defined. It does imply that timing plays a role, and this is explored later in this article.

What Do the Regulations/Guidances/etc. Say?

As mentioned above, I did not find any regulations or guidances that defined when to start capturing AEs/SAEs beyond those potentially associated with study treatment. Here is what I did find that relates to this discussion.

ICH E2A Clinical Safety Data Management: Definitions and Standards for Expedited Reporting

Focuses on the requirements for expedited reporting, rather than on determining the timing of what should be captured
Lists the minimal information necessary to send a report, which includes a treatment having a plausible causal relationship with the event
States that there are circumstances where certain SAEs may be exempt from routine reporting, such as when the SAE may be the primary outcome and expected. (KH: This indicates that there is no absolute rule that all SAEs must follow expedited reporting, although this does not speak to whether they are captured.)

Definition: Adverse Event (from ICH E2A):

Any untoward medical occurrence in a patient or clinical investigation subject administered a pharmaceutical product and which does not necessarily have to have a causal relationship with this treatment. An adverse event (AE) can therefore be any unfavourable and unintended sign (including an abnormal laboratory finding, for example), symptom, or disease temporally associated with the use of a medicinal product, whether or not considered related to the medicinal product. (emphasis mine)

Different regulations/guidances have slightly different wording, but all require a potential association with treatment, so if the event occurred prior to treatment it cannot, by definition, be an AE. Thus, a sponsor could capture only those AEs beginning with study treatment and be technically completely compliant.

Definition: Serious Adverse Event (from ICH E2A):

During clinical investigations, adverse events may occur which, if suspected to be medicinal product-related (adverse drug reactions), might be significant enough to lead to important changes in the way the medicinal product is developed (e.g., change in dose, population, needed monitoring, consent forms). This is particularly true for reactions which, in their most severe forms, threaten life or function.

This is part of the definition of an SAE, and indicates some reasons why expedited reported should happen for SAEs. As SAEs are, by definition, a kind of AE, they too cannot occur prior to treatment initiation. In reality, there are times when treatment may not be the only factor to consider, especially with respect to SAEs.

ICH E3 Structure and Content of a Clinical Study Report

5.3 PATIENT INFORMATION AND CONSENT: How and when informed consent was obtained in relation to patient enrolment, (e.g., at allocation, pre-screening) should be described.
10.1 DISPOSITION OF PATIENTS: (…) It may also be relevant to provide the number of patients screened for inclusion and a breakdown of the reasons for excluding patients during screening, (…)

Item 5.3 suggests that IC and enrollment are not necessarily the same time point, and 10.1 reminds us that subjects can be excluded during screening, prior to treatment.

ICH E6 Consolidated Good Clinical Practices

In section 6.9.2, it states that the clinical trial protocol should include The number of subjects planned to be enrolled. In multicentre trials, the numbers of enrolled subjects projected for each trial site should be specified. Reason for choice of sample size, including reflections on (or calculations of) the power of the trial and clinical justification.

This implies that enrollment refers to the subjects needed for analysis, not those screened.

21 CFR Part 312.62

(b)Case histories. An investigator is required to prepare and maintain adequate and accurate case histories that record all observations and other data pertinent to the investigation on each individual administered the investigational drug or employed as a control in the investigation. (…) The case history for each individual shall document that informed consent was obtained prior to participation in the study.

The regulation does not define what “participation” means, but since signing IC means agreeing to participate in the study, anything that happens after IC is, by definition, participating in the study. It’s a bit circular, but there it is!

ISO 14155.2 Clinical Investigation of Medical Devices for Human Subject – Good Clinical Practice

Definition 3.32: Point of Enrollment - time at which, following recruitment, a subject signs and dates the informed consent form

This brings up an interesting point:

Patients sign IC prior to participating in the study, meaning prior to any study procedures being performed (including those for screening)
Screening procedures are performed both to determine eligibility and, when appropriate, establish baseline values
If eligible, the subject continues. It is only at this point that the subject can be considered to be “on study.” Prior to this, eligibility has not been established, and so the subject must not be in the study, as GCP does state that ineligible subjects should not be included! (ICH E6 4.5.1 & 6.5.1)
Randomization may or may not occur at this point, depending upon the study design (there may be washout, baseline observation or other periods prior to randomization).
All of which means that the ISO definition of enrolment does not follow the same logic as the other regulations and guidances.

Even here it’s not completely clear, because later in the same standard, the Monitoring requirements state that the monitor should verify that:

f) signed and dated informed consent forms have been obtained from each subject at the point of enrollment and/or before any clinical investigation-related procedures are undertaken,
g) only eligible subjects as defined in the CIP are enrolled in the clinical investigation,

If only eligible subjects are enrolled, and enrollment happens at the time of IC, then all subjects who provided IC must be eligible, which cannot be true because screening procedures have not yet begun!

FDA’s Detailed guidance on the collection, verification and presentation of adverse reaction reports arising from clinical trials on medicinal products for human use, April 2006

5.1.1.2 Other safety issues requiring expedited reporting Other safety issues also qualify for expedited reporting where they might materially alter the current benefit-risk assessment of an investigational medicinal product or that would be sufficient to consider changes in the investigational medicinal products administration or in the overall conduct of the trial, for instance: (…)
c) new events related to the conduct of the trial or the development of the investigational medicinal products and likely to affect the safety of the subjects, such as: - a serious adverse event which could be associated with the trial procedures and which could modify the conduct of the trial, - a significant hazard to the subject population such as lack of efficacy of an investigational medicinal products used for the treatment of a life-threatening disease, (…)
5.1.2 What should not be reported? Expedited reporting is not usually required: - for reactions which are serious but expected, - for non-serious adverse reactions whether expected or not. It is generally not necessary to report events that are considered unrelated to the investigational medicinal product.

This last part speaks particularly to the distinction between capturing the SAEs and doing the expedited reporting. Section c. above brings in the point that study procedures may also cause SAEs, and although SAEs by definition must occur during treatment, study procedures happening during pre-treatment periods may also be important.

The take-home message is that one has to think about the context of the definitions, the spirit of the regulations/guidances/etc., and the specifics of the trial in order to determine the best course.

The Importance of Context

Why do we capture AEs and SAEs in a trial? We are trying to determine if the treatment causes undesirable effects that outweigh its benefits. We want to tease out the relevant events from the “background noise.” If randomization has happened appropriately, and there are no other sources of treatment assignment bias, and the investigators understand how to identify treatment emergent AEs, then the incidence of any given non-treatment-related AE in each treatment group should be the same, as should the incidence of the AE in the pre- and during-treatment periods. This suggests that, absent any additional risk factors or protocol design requirements, it should be unnecessary to capture AEs prior to treatment.

The question then becomes what to capture and/or what should follow expedited reporting rules. The following are other factors to consider.

Screening procedures: if the screening procedures are invasive or otherwise risky, capturing AEs/SAEs prior to randomization may be desirable, as it may influence the conduct of the trial, and/or the requirements for patient monitoring after the product is approved. Whether they should be subject to expedited reporting would depend upon an assessment of the other factors below.
Indication/population: how severe is the indication? If the subject population is quite ill, and SAEs are expected, then it may be appropriate to capture the SAEs, but not do expedited reporting. This should be defined a priori in the protocol after discussion with regulatory authorities, the company’s regulatory affairs and clinical colleagues.
Time frame: If the decision is to capture SAEs for subjects who were randomized but never receive treatment, what time frame should be used? Should they be captured only for subjects who didn’t receive treatment because of the SAE? Should the subjects be monitored for the same follow-up duration as treated subjects? Should the SAEs be subject to expedited reporting, considering that it is known the subject was not on treatment?
How much is known? If very little is known about the indication, treatment, study population and/or expected SAEs, then it would be appropriate to capture and report more information, as it is more likely that an event would be unexpected. As noted above in the FDA’s AE reporting guidance Section 5.1.2, if certain SAEs are already known to occur and this is documented in the Investigator’s Brochure, expedited reporting may be unnecessary, even if they are still “reported” on the CRF.
Geographic location: where is the study being conducted? The regulatory authorities in different regions may have different requirements or preferences for how much should be captured/reported.
Study design: randomization does not necessarily happen when the subject is enrolled, meaning that IC is signed and all eligibility criteria are met, and the subject is cleared to continue in the study. There can be washout periods, baseline observation periods, or other epochs that occur prior to randomization, and collecting AEs and/or SAEs may or may not be necessary. Much depends on the procedures performed and whether there is interest in comparing AE/SAE incidence before and after treatment.

Consequences of Reporting and Not Reporting

There are other, perhaps less obvious, consequences to these decisions.

Capturing SAEs: capturing SAEs requires providing considerably more information than is necessary for AEs. This is an additional burden on the site. It can also be a burden and expense for the sponsor, as it requires additional attention at each stage from data entry to data management to analysis and report writing.
Reporting SAEs: The burden is even greater when the expedited reporting processes are followed, as this requires completing additional forms, informing the sponsor and also the Institutional Review Board (IRB)/Ethics Committee (EC). The IRB/EC can be swamped with reports of routine events, which reduces their effectiveness. Finally, the receiving regulatory authorities have to distinguish between important events and those that are reported because the site/sponsor/etc. is being ultra conservative. This impairs their ability to respond to the critical events.
Analyzing the data: analysis and study reporting happen using the data that are captured. Whether SAES from randomized-but-not-treated subjects are included in the general intent-to- treat analyses or are put in a separate table is the choice of the biostatisticians, medical writers and clinicians responsible for the study report.

Bottom Line

I don't think it's possible to answer the overall question with an absolute statement of yes or no.

It is usually appropriate to capture SAEs for subjects who have been randomized but not treated, but they generally don’t need to follow expedited reporting.
It’s also usually unnecessary to follow expedited reporting for SAEs for subjects who have signed IC but have not been randomized (assuming randomization is when they are enrolled), but whether to capture them is more ambiguous.
Generally, it’s not required to capture AEs for subjects who signed IC but weren’t randomized.

The caveat is that there are exceptions to every one of these cases, as suggested by the earlier discussions. Like so much of what we do, the decision requires judgement. Regardless of your decision, it is good practice to define clearly in the protocol what is meant by “enrolled,” “on study” and any other similarly unclear term.

In order to make the right decision,

Educate yourself on the variables involved, and read the regulations
Gather the relevant questions and information
Talk to the other functional areas – this decision cannot (MUST NOT!) be made in a silo, because other perspectives and knowledge bases are required to ensure that all angles are covered

Your Regulatory Affairs group may have already spoken with the regulatory agencies about this; a good time to bring it up is at the early Phase 2 or end of Phase 2 meeting with the regulators. Whatever decision is made, be sure that it is documented fully, including the rationale for each element, because you may need that documentation later to justify the decision. There are few absolutes in our business, and nowhere is that more true than when dealing with regulations.

Many thanks to those who posed and responded to the question on LinkedIn. The thread was accessed on 6 December 2009 at http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers=&discussionID=10132121&gid=7 7402&commentID=8843270&goback=.anh_77402&trk=NUS_DISC_Q-subject#commentID_8843270

Disclaimer: The author does not work for a regulatory authority, and the material in this article is based on reading the regulations/guidances and applying her own experience and observations. Each company should confer with their internal experts and the appropriate authorities to determine the best approach for their situation.

Structuring Clinical Data: AE Seriousness Fields

2009-10-18T07:22:00.021-05:00

Even in a world with data standards, clinical data can be structured in different ways and still be standards-compliant. This is especially true when the standards define what to collect, but not how, or when, or in what combination. This article takes one example, the serious adverse event (SAE) fields, and explores several designs and the circumstances in which they could be appropriate. All examples are CDISC-compliant, but are not necessarily compatible with each other, emphasizing the need for careful thought in implementation.

Premise

Most people, when designing an adverse event (AE) case report form (CRF), believe they know the right way to capture AE Seriousness data. A random selection of 10 people would produce at least 3 different opinions, each of which can be best practice in a specific situation. Implement a design based on one set of assumptions in another environment, however, and best practice can become seriously flawed. This article explores different data capture designs for AE Seriousness and discusses where they would be appropriate.

Defining “AE Seriousness”

The US regulations and the ICH guidances are uncharacteristically clear about what constitutes a serious adverse event, and they are consistent with each other. The ICH definition of “Adverse Event” is

Any untoward medical occurrence in a patient or clinical investigation subject administered a pharmaceutical product and which does not necessarily have to have a causal relationship with this treatment.

A serious AE is a one that meets the above definition and additionally has one or more of the following characteristics:

Fatal Life-threatening
Requires in-patient hospitalization or prolongs hospitalization
Results in significant or persistent disability/incapacity
Congenital anomaly/birth defect

“Overdose” was originally on the list but was removed some years ago. Medical device guidance adds “serious injury/serious illness”, which is defined as “necessitates medical or surgical intervention to preclude permanent impairment of a body function or permanent damage to a body structure.” There can also be additional AEs that are serious for a given trial based on requests from the regulatory authorities. SAEs must be reported promptly to the regulators, and the reports include many details that are not entered into the clinical database. These may include hospitalization, the circumstances of death, autopsy reports, and a full written narrative of the case. The clinical database generally treats SAEs and AEs the same, other than SAEs bring flagged to indicate that they are serious.

Storing Serious AE Data

Historically, while non-serious AE data were stored only in the clinical database and managed by the clinical data managers, SAEs were often handled by the sponsor’s regulatory or drug safety group. As CRFs were often not completed for weeks or months after the subject’s visit, this enabled the required expedited regulatory reporting and supported ongoing safety monitoring. Although electronic data capture and reporting systems are more available, and there are a few integrated solutions, CRF data are still generally not entered real-time, and most organizations still store this information in two different systems, each of which may contain partial data. While it must all be included in the AE domain for an SDTM-compliant submission, that merely means that it must be available, not that it must be stored in one place.

Some Considerations

Current industry standards (such as CDISC’s CDASH and SDTM) provide two approaches to recording in the clinical database whether the AE is serious – a single yes/no field, and a list of fields defining the serious criteria that are each marked either with a yes/no question or a “check all that apply” structure, as shown here.

Recommendations

The standards do not specify how to use the fields or how they should interact. Here are some things to consider when making this decision.
Flagging AEs as serious in the clinical database is necessary. It allows the AEs to be summarized separately in the study report, and identifies those that must be reconciled with the safety database. Generally, CDASH discourages capturing the same information twice, which implies that the form should either ask if the AE was serious, or capture the list of criteria.
CRFs, whether paper or electronic, are usually not completed during or promptly after the subject’s visit, making them a poor trigger for the SAE reporting activities, and suggesting that the individual criteria list should not be used for that purpose.
Generally, especially for shorter lists, individual yes/no questions are preferable to “check all that apply,” especially when it is critical to evaluate each item. This is consistent with the CDASH recommendations. Although there is no firm cutoff number above which “check all that apply” is acceptable, the decision should be driven by the importance of evaluating each item and how much additional CRF completion work would be created by individual questions.

Following are some different options and the circumstances in which they would be appropriate.

Situation	Field inclusion
The safety database captures the serious AE information including the serious criteria and The seriousness criteria are readily available to the summary programmers	Clinical database should contain just the one yes/no field
Paper CRFs are used	The clinical database should contain either the one yes/no field (if the criteria are available from the safety database) or the individual criteria fields, but not both
The safety database does not contain the individual criteria or The data are not readily available to the summary table programmers	Clinical database should contain just the individual criteria fields
The clinical and safety databases are integrated so that data capture is performed only once	Clinical database should contain the individual criteria fields
The EDC system uses the one yes/no question to determine whether to bring up the individual criteria questions	Clinical database can contain both sets of fields as long as entry checks ensure data consistency
The EDC system asks the individual criteria questions and if any is “yes”, the one yes/no question is derived to “yes”. This may be used to trigger access to the safety database, but this approach should only be used if there are system requirements or some other compelling reason to do it.	Clinical database can contain both sets of fields, but only the individual criteria fields should be entered

Conclusion

As suggested at the beginning of this article, there isn’t one correct answer to the question of what fields to include. That decision needs to be made in consultation with multiple functional areas to ensure all needs are met, and requires the application of judgment. Like so much of clinical data design, the more thought and collaboration there is, the higher the quality the result.

How Bad Does It Have to Get?

2009-09-29T21:05:00.006-05:00

A recent study in PLoS One by E. Ray Dorsey, et al, found that between 1995 and 2005, the money spent on biopharm and medical device research has increased hugely across the board, and by as much as 369% in at least one therapeutic area. We have not, however, seen a comparable rise in the number of drugs approved by the FDA. Although the authors readily admit that their conclusions are based on a “sample of convenience,” they ring true to those of us in the industry. Whether you think that new drugs cost $600M or $1.2B to bring to market doesn’t really matter – we have thrown ever more money at a constantly diminishing return.

The authors suggest that now the limiting factors are non-financial, and solutions include bringing people and ideas together in more productive ways, bringing down the cost of clinical trials (though they don’t explore what this would take), openly disseminating the results of negative trials, and increasing the flow of people and ideas between organizations.

These all point to the need to revise and harmonize our processes, break down the functional area silos, increase the intelligent standardization of data and processes, and develop approaches that are nimble, flexible and adaptable. We need greater transparency in regulatory discussions and decision-making, which will help us to interpret regulations better and stop being so ultra-conservative in our work. Most of all, we need to understand that breaking the work into ever smaller pieces and parsing them out to ever more internal and external groups guarantees that no one sees the full picture, and so no one can control it. All of these changes must happen not only within organizations, but across them as well, to result in the best possible solutions.

The current approach is both failing and unsustainable. Neither government nor the public will tolerate the current mix of risk, benefit and cost. Unless our clinical trials enterprise changes dramatically, we will find ourselves forced to lower the prices we charge while maintaining the current costs. That is not a future that we, as future patients, should support.

The referenced study is at http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0007015
Photo: Dwarf ginseng, reputed to have calming medicinal properties. Taken at Sharon Hollow Nature Preserve, April 2007.
© 2009, Kit Howard

Upholding the Scientific Method

2009-09-21T09:53:00.006-05:00

Many thanks to Mark Uehling, from ClinPage, who distilled my recent rants to their pure essence, namely:

By not sharing our clinical trial operational processes, we are violating the most basic tenet of scientific research, which is to share methods and data in order to progress. Without this, we jeopardize the future of our entire enterprise.

Amethyst crystal embedded in clear quartz, photo from www.wilenskyminerals.com

2009-09-15T10:01:00.005-05:00

For those of you reasonably familiar with my fairly wordy style of writing… Hmm. That doesn’t do it. It’s usually good to learn new things, and recently I was made aware that… No, I don’t think that’s it either. OK. Here goes. I’m practicing being less wordy!

That was hard! My philosophy has always been “write it down, then edit to 50% of the size.” While editing is still good, I don’t have the bandwidth for that much rewriting, so I’m trying something new. This blog will still be a space for observations and rants, but my goal is to practice getting to the point quickly. Maybe in time it will become a habit! So here goes.

Recently I learned about OpenClinica, a free open-source clinical trials software system from Akaza Research. It apparently has been used in many trials by government, academia and in the industry. They join a growing trend of organizations no longer willing to accept industry’s rules (excuses?) of high costs and selective pipelines where treatments are developed only if they match a marketing strategy. Organizations like OneWorld Health, founded by Victoria Hale (one of my very few heros), a non-profit drug company that develops treatments for third world diseases. Or like the Parkinson’s Disease Society, that funds research to the tune of £4M per year.

Granted, these organizations have an advantage. They often receive their compounds from companies that have found them and don’t want to develop them, so much of the cost and risk of the initial research is mitigated. They generally don’t have shareholders demanding high profits. But they also don’t expect 7, 8 or 9 figure salaries, and it seems that their primary goal is treat patients, whereas much of industry seems to regard that as second to profits and personal wealth.

Most of us want to do the right thing, though, and one way is reconsider what we keep confidential and what we share. No, not chemical formulae or marketing strategies! I’m talking about clinical trials processes. How we develop our protocols, and SAPs, and CRFs. How we recruit our subject, monitor our data, resolve our queries. What we query. How we pass data between functional areas. There are efficient and inefficient ways to do these things, and we’ve been reinventing them for 20 years as new companies start and existing companies reorganize. It’s time for everyone to be open about how they do these things, and let the best approaches bubble to the top. And don’t tell me this is part of competitive advantage – no one out there is getting drugs to market dramatically faster than anyone else!!! Here’s a radical concept – let’s compete on the merits of the treatment or device itself!

Photo: St. Johns Wort, Auckland Botanical Gardens, Auckland, New Zealand. c. 2008, Kit Howard.

An Industry Evolving. Or not.

2009-06-07T23:33:00.014-05:00

An updated version of the entry that was here can be found at http://www.clinpage.com/article/a_deep_process_problem/C12

Not My Job???

2008-11-25T23:31:00.009-05:00

There is something I just don’t get about the whole contracting-out-data-management thingy. Sometimes it's done because of lack of resources, or lack of internal expertise, which is understandable and part of a flexible business model. Sometime, though, organizations look at data management and decide it is not a core competency, and that's why they outsource it or offshore it, or whatever the flavor du jour is, and that is the part I don't get. Many don't stay with only one CRO, and the studies are spread around. Often the protocol is tossed over the wall, and the CRO is expected to develop the database, cleaning rules, CRFs, etc., using their own approaches. Sure, the sponsors review the specs that the CROs produce, but either don't have the time or else the expertise in-house to assess them for relevance, completeness, and consistency with other vendors in the study, let alone poolability with other studies in the development program. If data are not consistently defined and captured, then pooling them may not be valid, and thus conclusions drawn from the data may not be valid.

Data management is about the design of the data, about its definition and assurance of its quality, with quality meaning "fit for its intended purposes." It’s about knowing how to capture data efficiently and accurately, and in accordance with the word and spirit of the protocol. Without the data, the study drug/biologic/etc is just another pile of white powder (or whatever) sitting in a jar. The data are what tell the story of the safety and the efficacy of the treatment. The data are the basis for the label, for marketing, for answering questions from the FDA and the public, and for finding new potential indications. The data are the foundation of the whole thing.

So help me understand.

What is it about this whole process that
ISN’T COMPLETELY ABOUT THE DATA??????

How on earth can data management be a non-core competency!?

Photo: the world at night, from orbit. NASA website.

Can You Create Well-Designed Consistent CRFs for the Site?

2008-07-15T19:22:00.007-05:00

It's all about perspective...

The title question was posed at the recent SCDM Data Quality webinar. Most respondents answered “yes”. My answer is “it depends”. From a given sponsor’s perspective, the answer is “yes”, but from a site’s point of view, it is definitely “no”. Sites may never do more than one study with any given sponsor, so from their perspective, the CRFs (or EDC system) is never seen again. Each set has slightly (or very) different questions, with different answers, grouped on different pages, and with different completion instructions. Visits have different names, forms for non-completed visits may or may not have to be returned to the sponsor, and entry edit checks fire for different reasons. If sites don’t keep these rules straight, we think they produce poor quality data.

So what is the solution? The CDISC CDASH project will resolve some of these dilemmas. It identifies the minimum set of data fields for most common study designs, along with CRF completion guidelines. Many fields are linked to standard terminology, ensuring that code lists are consistent. This goes encourages similar content, and that will help the sites, but a major obstacle remains. On the whole, each company’s data management group believes that it has the best answer to each of these challenges; it has developed the best and highest quality solutions, practices and procedures, and few have any interest in changing. If one assumes that most have had submissions accepted then one of two possibilities must be true. They are either all “good enough” for the purposes of the development project, or all the QC, QA, audits and oversight are so sloppy that they fail to detect the flaws in these processes. Granted, there are differences required by some study designs, indications, and drugs/biologics vs devices, but my experience suggests that these are very minor.

That leads to the inevitable conclusion that these variations are a matter of preference, and do not impact quality. Some will be more efficient, precise, or suited to habit, but they achieve the same result. Think about what happened the last time you were shifted to another project in your organization. Chances are that you had to learn new rules, and until you learned them, you were more likely to make mistakes. There is no reason to believe it is any different for the sites.

Why, then, can’t we agree upon common practices and rules and approaches for these common activities? Are we so convinced of our own superiority that we refuse to change? Are we afraid that others will steal our good process ideas and get to market first? Are we just “used to doing it that way” or have “always done it that way” or believe that “regulations or Biometrics or Clinical require that we do it that way”? In other words, if it ain’t broke, don’t fix it? Well, I argue that it is “broke”. It is “broke” for the sites, and if it ain’t fixed we’ll continue to lose investigative sites, and pour ever more resources into trying to inspect quality into the data, and miss the opportunity to maximize the CDASH revolution.

So what do you think? Do you agree? Disagree? Want to throw turnips? Should life be more consistent for the sites at the expense of our processes? We’re a pretty inventive bunch – I bet we could find ways to be efficient while collaborating with the sites to improve consistency.

I’ve started a list of practices that I think could be harmonized. Do you agree with the list? What can you add? What shouldn’t be there? How do you approach these activities and why? Are your reasons based in concrete need or historical habit or the belief that someone else internally “won’t like it”? How can we create a sponsor/site forum that would be trusted by both groups? Who might have to collaborate internally and externally to make this happen? Let’s talk!

1. When a subject terminates early, do CRFs for all visits after termination have to be returned to Data Management? (applies to paper only, I assume)
2. Do sites complete new AE and Con Meds forms for each subject at each visit, or are existing forms updated? i.e., are AEs and Con Meds info captured in a visit-based style or a log-based style?
3. How should visits be referenced? Can they be standardized to 1, 2, 3, etc., with a possible variation for course-based studies (e.g., oncology, although that could probably be Visit 1, 2, 3 etc within each course)?
4. When should we start capturing AEs and SAEs? Informed consent? Start of treatment (technically it can’t be an AE if treatment hasn’t started)? SAEs at informed consent and all AEs at treatment start? Should it depend upon whether the study requires potentially harmful screening procedures? On something else?
5. Once the CDASH data fields have been finalized, can we agree on a consistent layout? What should it be for each of the domains?
6. For Adverse Events forms, should the layout be portrait with one AE per page or screen, or landscape with multiple lines per page or screen? What are the pros and cons of each? Note that this is not talking about how they are stored in the database – just how the site would see them.
7. Should monitoring guidelines be given to the sites? Put in the site’s study manual? Included in the EDC application? If not, why not?
8. Should the list of edit checks be given to the sites? If not, why not? Don’t we want the sites to understand what the data should look like? If so, how can they be presented so that they are accessible and understandable?

Each of these, and many more, can be discussions in their own right, and the answers to each depend upon any number of assumptions about the underlying processes, but surely, if we really want to, we can make this work. I look forward to your comments.

Photo: Romulus, Ann Arbor, Michigan. c. 2008, Kit Howard.

Who Are You?

2008-03-20T21:50:00.002-05:00

The other day I heard the old song by The Who “Who Are You”. It has been a while, and I can’t follow most of the lyrics on the track, but it doesn’t really matter. What struck me, as I was thinking about our business, is “Who Are You?” We each have a number of roles we play.

We are Pharmaceutical Professionals. We might be a monitor, or clinical scientist, or programmer, or data manager, or any one of a plethora of other roles. In those roles we look at scientific puzzles, and try to ask the best questions to clarify the unknown to formulate the quickest reasonably concrete result. In that world we have control – we are the scientists, the providers of answers, the managers of data, the spinners of results. We tend to think of our day-to-day existence as a set of tasks. This database must be closed. That report must get written. The other article must get published so the following ad will appear “true”. This is how we earn our living, how we pay our mortgages, how we pass the time...

We have other roles as well. Roles that only surface when things go wrong. Our parents become ill, and suddenly we are trying to find out whether the medication or combination of medications prescribed for them are likely to help and/or have intolerable side effects. Where can we find the information? Some of it is on WebMD, some in the Merck manual, some perhaps in our own company data banks, some we get from colleagues in other companies. We discover that the information is fragmented, hard to come by, spun to minimize the negative and maximize the positive.

There is often no way to tell what subsets of patients are most likely to benefit, either because the information is not published, or it is published in scientific journals not available to the general public, or it is buried in reports so dense that x-rays could not penetrate them. Or maybe the information simply doesn’t exist – no one has ever done a study to see if that particular combination of medications in that particular combination of co-morbid conditions is beneficial or harmful or ineffectual…

So where can we turn? The pharma companies, our employers, don’t have any incentive to do these studies. The government bodies such as the NIH have their own research agendas. Who can tell us? In fact, given that there is an almost infinite number of possible combinations of medications and co-morbid conditions, can we ever hope to have any solid evidence? Maybe we have to make do with fragmentary data extrapolated from studies that don’t really reflect our situation in an environment where the optimal treatment may be unidentifiable and in any case would not be covered by insurance.

Then, eventually, we become the patients. Not subjects, for those are the ones participating in the trials that may or may not provide any benefit for them as individuals. The patients. Ill, disabled, frightened, tired, without the energy or mental strength to do the research to make sure that the treatments we are prescribed (by the doctor who may have graduated bottom of the class) are not going to worsen some other condition we have, or perhaps are just the usual run-of-the-mill treatments that don’t incorporate the latest findings.

As we get older, more and more of us join the ranks of the patients. Then we retire, and find that not only are we the patients, we are also the marginalized elderly, those who don’t work, who don’t contribute, who are seen as a burden, whose “entitlements” take an ever larger chunk of the federal government’s budget. As the baby boomers grow older, and the generation following is dramatically smaller, who will pay for our aging?

It is very clear to me that we have no choice. We have a moral responsibility to our elders, to ourselves and to our children to fix this system. We have to make drug discovery, development and marketing not only faster and less costly, but even more importantly, dramatically more transparent and more driven by the needs of the patients. It may not be immediately apparent how this can be compatible with corporate goals, but I submit that if we do not find a way to satisfy both imperatives, we as employees, we as caregivers and we as patients will be among those who suffer the consequences.

The Blossoming of Standards

2007-09-14T10:40:00.001-05:00

It recently came to my attention that the National Institute of Neurological Disorders and Stroke has been conducting a project to define a data elements dictionary that can be used in trials it sponsors. The intent is both to facilitate data collections and storage as well as to promote data sharing. While there are many similarities in the data needed for different therapy areas, there are also many differences in both content and in structure. This makes creating a single data dictionary something of a challenge.

Tackling this challenge would be a tall order in a company, but there everyone (at least in theory) reports into the same organization, and, provided management has the intestinal fortitude to do so, common approaches can be mandated. The NINDS project has an even greater challenge, in that it has a significantly smaller degree of control over the structure of the studies it sponsors and thus must provide carrots rather than sticks (i.e., give investigators incentives to cooperate, rather than force them to do so).

This project is an example of an increasingly popular trend towards sharing data in the academic and government research environments, and to a lesser extent in industry (the latter being driven mostly by the need to create consistently structured data for regulatory submissions). CDISC is the most familiar one to those of us in industry, and it is aimed primarily at regulatory agencies. Outside of industry, I know of HL-7, which is (among other things) a set of messaging standards for defining and exchanging data between healthcare providers. There is also the National Cancer Institute with its caBIG project, which is a set of data dictionaries and processes for standardizing the definition, collection and storage of data from oncology trials sponsored by NCI. There may well be other projects.

It is wonderful to see such awareness of and enthusiasm for standardization. This can only help to increase efficiency, reduce costs, and enhance the safety and efficacy of medical treatments. On the other hand, it is a concern that, although these organizations do have some awareness of each others’ work, they are still working mostly independently. Not only is there some duplication of effort, but they will inevitably make different decisions and produce slightly different results. Anyone involved in such a project understandably has a tremendous sense of ownership and will be committed to his or her own decisions. At some point the various schemas will have to be harmonized, and this will require a great deal of work both to achieve the harmonization and then to apply the results back to the source systems. Because of this, it is critical that the organizations already developing standards be open, transparent and vocal about their work so that any other organizations contemplating initiating a standards project will join an existing one and work toward ensuring their needs are met there. By the same token, it is encumbent upon the existing projects to be open to the needs of others so that the number of projects is limited and eventual integration is not even more daunting.

Photo: Phyllachne colnsoi. Haast Pass, South Island, New Zealand. Flowers approx 3 mm diameter. c. 2006, Kit Howard.

The Washington Monument was Built by Aliens!

2007-05-30T18:50:00.001-05:00

It has been said that figures don’t lie but liars figure. In his book Fads and Fallacies (1) Martin Gardner presented this cautionary tale as part of his examination of the numerical myths associated with the Great Pyramid in Egypt.

… If one looks up the facts about the Washington Monument in the World Almanac, he will find considerable fiveness. Its height is 555 feet and 5 inches. The base is 55 feet square, and the windows are set at 500 feet from the base. If the base is multiplied by 60 (or five times the number of months in a year) it gives 3,300, which is the exact weight of the capstone in pounds. Also, the word "Washington" has exactly ten letters (two times five). And if the weight of the capstone is multiplied by the base, the result is 181,500 —a fairly close approximation of the speed of light in miles per second. If the base is measured with a "Monument foot",(2) which is slightly smaller than the standard foot, its side comes to 56½ feet. This times 33,000 yields a figure even closer to the speed of light.

And is it not significant that the Monument is in the form of an obelisk—an ancient Egyptian structure? Or that a picture of the Great Pyramid appears on a dollar bill, on the side opposite Washington's portrait? Moreover, the decision to print the Pyramid (i.e., the reverse side of the United States seal) on dollar bills was announced by the Secretary of the Treasury on June 15, 1935—both date and year being multiples of five. And are there not exactly twenty-five letters (five times five) in the title, "The Secretary of the Treasury"?

It should take an average mathematician about fifty-five minutes to discover the above "truths," working only with the meager figures provided by the Almanac.

One definition of the quality of data is its “fitness for use”, which implies that the use has been defined. There is increasing interest at the FDA and within many companies in establishing data warehouses that will allow for meta-analyses that are orders of magnitude bigger than any possible today. We must be careful that, in our haste to mine the data for interesting relationships we do not use the data in ways that were never intended and create our own mythologies.

(1) Martin Gardner, Fads & Fallacies in the Name of Science. Dover Publications, New York.
c. 1957. pg 179.
(2) Derived by dividing the length of one building stone by 25, i.e., five times five.

Absence of Evidence...

2007-04-24T18:24:00.001-05:00

In archaeology there is a popular phrase that says that “Absence of evidence is not evidence of absence.” In essence, it means that the fact that nothing was found does not mean that nothing exists. It merely means that it wasn’t found. For example, let’s assume that I conduct an archaeological dig in my back garden in Michigan and I find no indication that there was a Potawatomi settlement there. There are those who would then claim that these Native Americans did not live here, whereas all that can legitimately be stated is that I did not find any evidence of them. In fact, given the topography and location, it’s quite likely that there was at least a seasonal camp nearby, but based on my research I cannot make an accurate statement one way or the other.

A similar issue exists in drug development. There has been a long-standing dispute between Clinical Data Management and most of the other functional areas involved in clinical trials that is centered around the use of “Not Done” and “None” data fields. A “None” box is completed on the Case Report Form (CRF) when there were no findings of a particular sort for a study subject. This is typically used to record that no adverse events were observed, or that no concomitant medications were taken. Similarly, the “Not Done” box is checked when a test or evaluation was not done. This avoids leaving a blank on the CRF, which would usually be queried by the monitor or the data manager.

There has been significant resistance to using these data fields. They are administrative fields, in that they do not contain data that are analyzed, and given that every additional field collected increases the costs of the trial, they would seem to be good candidates to drop. In addition, the FDA has typically said that they want to see only the data that represent that something happened, and not lists of “Not Done” and “None” fields. The CDISC submission standards do not define “Not Done” fields, so organizations that are modeling their entire data stream on CDISC have nowhere to put these fields. Finally, the fields create opportunities for superfluous data queries when they have been marked but there are also data values present.

All of these are good reasons for dropping “None” and “Not Done,” but they are all trumped by the fact that if “None” and “Not Done” are excluded, we cannot say if the tests were not performed, or if they were performed but the site did not complete the CRF. These are obviously two very different situations. For example, an ECG CRF that records no findings will mean one thing if the test results were not transferred from the source documents, something very different if the test was not done, and something different again if the test was done and there were no abnormal findings. In the first and third cases, we can retrieve the results and know whether or not there were significant findings. In the second case, we can’t know one way or the other.

It is important to note that there is no reason why the presence or absence of “Not Done” and “None” should affect the structure of the submission, nor the data presented in tabulations and listings. Subjects who had no findings and those whose evaluations were not done can easily be suppressed from the display. The net effect is to increase the reliability of the displays, as reviewers can be certain that blanks and absent records indicate that the data are really not available, rather than just omitted. We can then be certain that, for the purposes of analysis and submission, the absence of data really is evidence of absence.

Photo courtesy of Archaeology Magazine, Archaeological Institute of America

The Missing Link in Clinical Data Standards

2007-03-30T08:53:00.002-05:00

The term “clinical data standards” means different things to different people. In the world of clinical trials, it has traditionally meant having case report forms and a database structure that are reusable from study to study. At the DIA CDM Annual Meeting last week in Orlando, Dr. Steve Wilson observed that, from the FDA’s perspective, the content, format and uses for data in regulatory submissions have evolved over the years. Initially, the push was to make the transition from paper to electronic submissions. With that, the need for structural standards became apparent, and hence the development of CDISC. We now recognize that the format of the data needs to be standardized, and CDISC is developing standard data collection modules (CDASH) and controlled terminology (e.g., code lists) to address this need. The FDA hopes to reap the benefit of this work in Janus, a data repository that will eventually house all submission data. This will allow them to monitor drug safety much more proactively.

These are all excellent developments, and bring us closer to an environment where standard data viewing and analysis tools are feasible and where databases and programs can be reused. I would argue, however, that there is one more step we need to take in order to be truly standardized. We are doing much to define the structure of data, but have very little in place to define its content. By this I don’t mean the terminology used to categorize data, but rather the processes and assumptions inherent in generating and collecting the data.

For example, suppose you want to analyze the emergence of adverse events in a particular drug class. You access the merged safety database, assign each AE to a time interval based on its start date, and then count the AEs and compare the intervals. It seems straightforward, until you realize that some studies started collected AEs when the informed consent was signed, some started at first dose, and some started collecting serious AEs when the IC was signed and all AEs at first dose. From the point of view of structure and terminology, the data are standard, but they are not suitable for this analysis, and while the collection starting point would be defined in the protocol, that information would rarely accompany the data.

This lack of definition about the processes and assumptions is, I believe, the greatest threat to data quality. In order for data to be truly comparable we must know not just its electronic characteristics but also the processes and assumptions made in generating it. Only then can we know when the data we have is fit to answer the questions we want to ask.