Innovator: Summer 2024

Putting Baystate Research on Sound Footing

The efforts of investigators and staff at Baystate Health have resulted in remarkable successes, including over 250% growth in our research program in the last 8 years and a record high of 290 peer-reviewed publications in 2023. The founding of the Department of Healthcare Delivery and Population Sciences has galvanized mission-driven research that will benefit our patients and community, and Chair Peter Lindenauer and Division Chiefs Elizabeth Peacock-Chambers and Kimberly Geissler have done yeoman’s work nurturing the next generation of Baystate clinician-investigators. The Clinical Trials Office was established in 2019 and has become a vital nexus for clinical research under the leadership of Gerard Coly and Viorika Nelson. In partnership with local nursing investigators, Cidalia Vital has revitalized nursing research, a key piece for our Magnet designation. We also continue to build on longstanding strengths in Critical Care, Heart & Vascular, Emergency Medicine and Oncology research. These are just a few examples, but with this growth has come growing pains and closer scrutiny of our ability to manage the expanding portfolio of studies. An apt analogy is that when building a house, you need to be sure the footings and foundation are sound before starting the framing.

A number of required research audits and remediations in recent years, including implementation of the OnCore clinical trials management system, brought to light complex operational, regulatory and financial management challenges. To help us diagnose and mitigate these issues, this past winter we engaged an external consultant, Huron Consulting Group,. Huron’s findings resulted in five main recommendations:

1. Re-establish and empower a research governance council & augment the Strategic Plan for Baystate Research.
2. Hire a senior leader for research operations.
3. Realign research infrastructure, and define roles, responsibilities, and accountability of research staff.
4. Streamline & enhance research workflows (including the use of technology and systems) and optimize the use of the OnCore Clinical Trials Management System.
5. Develop BH system-wide onboarding, training and education programming for research staff and investigators.

Given the scope of needed improvements, it became clear that it was necessary to institute a temporary moratorium on accepting new sponsored clinical trials or submitting new, non-NIH grants. This hiatus is providing “breathing room” for research staff to bring our ongoing studies and processes into better compliance with our policies and national standards. We have a high bar for exemptions to the research pause and have approved only a handful of exceptions on a case-by-case basis. We have also brought in external consultants as acting research leaders to help us implement the necessary changes: Shannon Chism joined us in late Spring as Interim Senior Director for Research Operations, and recently we welcomed John Sites as Interim Director of Sponsored Programs.

Baystate Health’s research community should be heartened by the high level of engagement of senior leadership in addressing these challenges. They remain fully committed to advancing and growing our clinical research enterprise to support Baystate Health’s strategic goals and to making the necessary investments to ensure we have a robust, effective and compliant research infrastructure. With the support of leadership and the research community, I am confident that the research pause and subsequent operational improvements will provide Baystate Health with the sound footing upon which to build impactful mission-driven clinical, health services and population research over the coming decades.

Peter D. Friedmann, MD, MPH
Chief Research Officer, Baystate Health
Associate Dean for Research, UMass Chan – Baystate

Finding Enjoyment through IRB Service: Meet the Members of the Baystate IRB

An Institutional Review Board (IRB) is a committee of research experts and community members that reviews research projects that study people. In the United States, all research involving people, their data, or their biospecimens must be approved by an IRB before it starts. IRB review ensures that human subject research meets high ethical standards and federal and local laws.

IRB members gain experience reviewing medical research. IRB service also offers an opportunity to hear about current research in the Baystate community. Members participate in ethical discussions relating to current studies and help represent the thoughts and concerns of the surrounding community.

IRBs must have at least five voting members with various backgrounds who can provide different perspectives on the research. Having a diverse composition ensures that research participants are adequately protected. Those members must include at least one who is not connected to the institution doing the research, and one who is not a scientist. Each IRB must have members who can review the specific types of research it oversees and members who know the community where the research takes place. Alternates and non-voting members may also be appointed, with alternates authorized to vote in the absence of the member for whom they are the designed alternate.

We welcome you to meet some of the IRB board members at Baystate and read about what they enjoy most about IRB service.

Robert Baevsky, MD, FACEP, Chair of IRB #2

Years of Service: 19
Why do you participate on the IRB and what do you enjoy most: I have been a principal investigator I and co-author on several research and published papers, giving me a researcher's perspective on the IRB. As I transitioned away from doing my own clinical research, I became more interested in the regulations around conducting research and, more specifically, those pertaining to human studies protection. As part of the IRB, I am able to help bridge the goals of the researchers with the many regulatory policies that exist. I enjoy getting an inside look at the various research projects that are taking place throughout the Baystate Health system. It is also fun working with a great bunch of knowledgeable IRB staff members.

Maripat (Mary P.) Toye, MS, RN, ACRN, CCRP, CPXP, Vice Chair of IRB #2

Years of Service: 11
Why do you participate on the IRB and what do you enjoy most: My experience in Pediatrics, Adolescent, Maternal Child and Family nursing provided me with a broad background in learning, knowing and meeting the needs and concerns in healthcare of families. Families taught me the importance of their cultural issues and their community needs. Safety, privacy, and protection in caring for patients and families within the health care system has always been a priority in my practice. I participate on the IRB as it gives me the opportunity to work within the research arena and learn about new research studies and issues. It gives me the chance to review a Protocol from the eyes of a participant and decipher if it is clear and understandable from a lay person perspective. I enjoy learning about the many diverse protocols and studies ongoing at BH. I learn about the new findings and advances and challenges in healthcare today. I enjoy and benefit from the collaborative nature of the IRB. It is made up of members from multidisciplinary perspectives and the community. The collegial and respectful exchanges of shared reviews and ideas gives me more insight into my own perspectives of clinical research.

William Soares, MD

Years of Service: 6
Why do you participate on the IRB and what do you enjoy most: As a researcher at Baystate who often works with vulnerable populations, I believe it is important that we do everything we can to maintain the safety and trust of our patients and research participant populations. I enjoy learning about the exciting new studies that our researchers at Baystate are working on every day, often behind the scenes, to improve the lives of our community members.

Mora Geoffrion

Years of Service: 10
Why do you participate on the IRB and what do you enjoy most: I joined the IRB team shortly after retiring from teaching. I am the nonscientist on the team, and that allows me to provide a different point of view from the scientists. I enjoy being able to learn about new studies and their potential benefits for very sick patients. As a mother, grandmother, and former teacher, I’m particularly interested in how participation in studies affects vulnerable children and their parents’ concerns. The team is very patient and respectful of my input.

Interesting in becoming a volunteer member of the Baystate IRB? Learn More About Volunteering

EpiBio Research Core (EBRC) Classroom: Compatibility Intervals

Alex Knee, MD, Assistant Professor of Medicine & Aixa Perez Coulter, MS, MPH, Assistant Professor of Surgery, UMass Chan-Baystate

Interpretation of 95% confidence intervals (CI) is one of the most difficult concepts to understand. Experienced professors lament over how to teach it; textbooks often teach it incorrectly, and most of us use them poorly. This is complicated by the fact that recent methodologic literature and journal requirements are reducing the role of the p-value in favor of the confidence interval. What are we to do?

If we go back to the basics, statistical inference is based on assumptions ranging from study design and data collection to analytic and model choices. We use statistical inference to estimate uncertainty using a sample from the population of interest. Traditionally, we use these assumptions to compute a P-value, also known as the probability of finding an effect at least as extreme as what was observed, assuming there is no difference. Unfortunately, we have developed bad habits, and all too often, a P-value is dichotomized into “statistically significant” (p-value is equal or less than the cut-off; usually 0.05) and “not statistically significant” (p-value is larger than the cut-off). However, a small p-value identifies the observed effect as being rare only if all the assumptions are correct, while a large p-value can be due to various factors, including no actual difference, failures in model assumptions, or not having sufficient power. In addition, when statistical inference is reduced to a binary decision, we are no longer quantifying uncertainty.

Enter the confidence interval as an alternative to the p-value. This method allows for a more nuanced interpretation of our observation. While all the assumptions listed above are still required, we can now expand our interpretation beyond a single number, which is even more important for evidence-based practice. However, another problem presents itself. How do we interpret a (usually 95%) confidence interval? If the p-value is a probability statement, is the confidence interval also a probability statement? Starting with the technical (non-intuitive) definition of a 95% confidence interval (upon repeated sampling, 95% of confidence intervals contain the true population parameter), we can see that probability is NOT part of this definition. If you want probability, then you need to use Bayesian methods (see Credible Intervals). Then what are we 95% confident in? This goes back to the theoretical concept of repeated sampling; we are 95% confident in the method only. If you conducted a study 100 times (or used repeated sampling), then 95 out of 100 intervals will contain the true estimate. However, this needs to be more intuitive as we rarely conduct repeated sampling, and we are left with only one confidence interval that either does or does not capture the true population parameter. These requirements confuse many researchers and make interpretation even more difficult for the general public. Ultimately, we want our confidence intervals to convey uncertainty. However, we have again taken shortcuts, and many researchers simplify the interpretation to significant (CI excludes the null) and not significant (CI crosses the null) instead of also interpreting the width (precision of the estimate) and clinical interpretation of the interval. This brings us back to the same problem, as there is no additional benefit in replacing the use of p-values with an oversimplified binary interpretation of the CI.

The literature presents several approaches to address this problem, but one of the most intuitive solutions gaining ground is the compatibility interval proposed by Rafi and Greenland (2020). They argue that the interpretation issues are not statistical; they are cognitive. The approach is simple (but mathematically sound): construct your interval using traditional methods. However, instead of using the complex language above, we can interpret the point estimate as the most likely effect size while estimates as low as (the lower bound of the CI) and as high as (the upper bound of the CI) are compatible with the data. We don’t have to worry about making incorrect statements about a 95% probability, nor do we limit our inference to a binary threshold. Even though both the P-value and confidence intervals can be considered a measure of compatibility, compatibility intervals are less prone to misinterpretation than the traditional P-value and confidence interval approach. We are now appropriately quantifying uncertainty! Using the label “compatibility” rather than “confidence” offers no false confidence in your results. This means changing our language from significant versus not significant (or 95% confident) to values of low compatibility and high compatibility to foster honest reporting of results. Rafi and Greenland discuss an observational study published in JAMA article by Brown et al. in which they report a 95% CI for a hazard ratio ranging from 0.997 to 2.59 and interpret this as a non-significant result. The authors concluded that no association existed as “statistical significance” did not exist. However, through the lens of compatibility, we can see that nearly all the estimates are compatible with an increase in the hazard rate! The compatibility interval encourages researchers and clinicians to spend more time interpreting study results before implementing changes in practice based on a binary interpretation.

This issue is prevalent in clinical journals. In January of this year, JAMA Neurology published an article by Zhang et al. in which they conducted an RCT among 601 patients with acute ischemic stroke, evaluating the use of argatroban and improved functional outcomes at 90 days. They observed an improvement in their primary outcome (modified Rankin Scale score [mRS] of 0-3) but no improvement in their secondary outcome of an mRS of 0-2 (see figure below). These are interpretations based on an arbitrary P-value threshold of 0.05. If we interpret the RRs using the compatibility framework, we can see that the secondary outcome is highly compatible with the primary outcome (RR~1.10 and overlapping CIs). Since the CIs are wider (less precision = more uncertainty = smaller sample size), we can conclude that a larger sample size should clarify the association. In addition, if we also evaluate the absolute risk differences for the primary outcomes, we can see that while the effect of argatroban may be statistically significant, the incidence of achieving an mRS score of 0-3 at 90 days may range from only a ~1% improvement (not likely clinically relevant) to a ~14% improvement (likely very clinically relevant). While the secondary outcome is slightly attenuated (and it is possible argatroban could cause harm), the results are still very compatible with the primary outcome as the CIs almost completely overlap. Attenuation should make sense as we are making the outcome more difficult to achieve (e.g., an mRS score of 0-2 vs 0-3). In addition, as fewer patients achieved the secondary outcome, we also have more uncertainty in the results (wider CI) than the primary outcome; more evidence that a larger sample size would clarify this association. A more nuanced interpretation would incorporate these compatibility estimates, not just report the presence of statistical association. Nothing magical happens at p<0.05!

We encourage you to pick up your favorite journal and critically interpret the results as compatibility intervals. We are less likely to make mistakes in interpretation, and chances are the author's conclusions may not actually be supported by the data. Remember, compute the confidence interval but interpret the compatibility interval!
Please reach out to us if you have any questions! We are also available at biostatistics@baystatehealth.org if there is a topic you would like us to cover in a future EBRC Classroom!

References and Further Reading
Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4):337-350. doi:10.1007/s10654-016-0149-3
Rafi Z, Greenland S. Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise. BMC Med Res Methodol. 2020;20(1):244. Published 2020 Sep 30. doi:10.1186/s12874-020-01105-9
Hawkins AT, Samuels LR. Use of Confidence Intervals in Interpreting Nonstatistically Significant Results. JAMA. 2021;326(20):2068-2069. doi:10.1001/jama.2021.16172
Savitz DA, Wise LA, Bond JC, Hatch EE, Ncube CN, Wesselink AK, Willis MD, Yland JJ, Rothman KJ. Responding to Reviewers and Editors About Statistical Significance Testing. Ann Intern Med. 2024 Feb 20. doi: 10.7326/M23-2430. Epub ahead of print. PMID: 38373303.
Xie, M., & Singh, K. (2013). Confidence distribution, the frequentist distribution estimator of a parameter: A review. International Statistical Review, 81(1), 3–-39. doi:10.1111/insr.12000
Hespanhol L, Vallio CS, Costa LM, Saragiotto BT. Understanding and interpreting confidence and credible intervals around effect estimates. Braz J Phys Ther. 2019;23(4):290-301. doi:10.1016/j.bjpt.2018.12.006
Gelman A, Greenland S. Are confidence intervals better termed "uncertainty intervals"?. BMJ. 2019;366:l5381. Published 2019 Sep 10. doi:10.1136/bmj.l5381
Brown HK, Ray JG, Wilton AS, Lunsky Y, Gomes T, Vigod SN. Association Between Serotonergic Antidepressant Use During Pregnancy and Autism Spectrum Disorder in Children. JAMA. 2017;317(15):1544-1552. doi:10.1001/jama.2017.3415
Zhang X, Zhong W, Xue R, et al. Argatroban in Patients With Acute Ischemic Stroke With Early Neurological Deterioration: A Randomized Clinical Trial. JAMA Neurol. 2024;81(2):118–125. doi:10.1001/jamaneurol.2023.5093

Office of Research says Goodbye

The Office of Research bids a fond farewell to two long-standing members of the team, Paul Visintainer and Melissa Quintero.

Paul Visintainer served as both the Director of Scientific Integrity and Analytics as well as the Director of the Epidemiology and Biostatistics Research (EpiBio) Core, retired at the end of May after nearly 16 years of service at Baystate.

Paul was a Professor of Medicine with a secondary appointment in the Department of Healthcare Delivery and Population Sciences at the UMass Chan Medical School-Baystate. Before joining Baystate, he was Chair of the Department of Epidemiology and Biostatistics at New York Medical College School of Health Sciences and Practice. With over 40 years of experience, he has collaborated with clinical researchers to provide expertise in study design, data analysis, and manuscript development. He has published more than 195 peer-reviewed articles and led a team of biostatisticians and epidemiologists who support resident, fellow and faculty research and research education. In addition, Paul has served across many committees, most notably as the founding Chair of the Institutional Scientific Review Committee and an Institutional Review Board Chair. We wish Paul the best in his retirement and thank him for his many years of service and dedication to improving the quality of research at Baystate.

Alexander Knee now serves as Director of EpiBio. Alex joined Baystate as a Clinical Research Coordinator 16 years ago. For the past six years, he has been the EpiBio Program Manager and was a Biostatistician on the team seven years before that.

Melissa Quintero, Director of Sponsored Programs Administration (SPA), departed Baystate Health to begin an exciting new role as Director of Sponsored Programs at AdventHealth Orlando.

Among her accomplishments during her 7 years at Baystate Health, Melissa reorganized SPA to a pre- and post-award structure, implemented processes to improve compliance with federal uniform guidance, and instituted procedures to augment funds capture on industry clinical trials. “While we are happy for Melissa's professional advancement and relocation closer to family in sunny Florida, we will miss her collegiality, knowledge of the regulations, and passion for research and supporting our investigators,” says Dr. Peter Friedmann. Good luck in your next chapter, Melissa!

MD/PhD Student Eric Romo presented the 2024 Chancellors Award from the Morningside Graduate School of Biomedical Sciences

Eric Romo received this year's chancellors award for his dissertation "The Influence of Spatial Proximity to Syringe Services Programs and Secondary Syringe Exchange on HCV Risk Among Rural People Who Inject Drugs." The award recognizes a graduating student who embodies the ideals of the biomedical research scientist through outstanding doctoral research, leadership, and service to the UMass Chan Community. It is the highest award presented to a Morningside Graduate School of Biomedical Sciences graduating class member.

For his dissertation work, Eric joined the Baystate-based field team in rural counties in eastern Vermont and western New Hampshire on a study of persons who inject drugs. “Our disenfranchised research participants quickly took to Eric, and he became a valued member of our multidisciplinary research team -smart, competent, detail-oriented, dedicated, compassionate, proactive, thoughtful, gentle, good-humored, and unflappable,” says mentor Dr. Peter Friedmann.

Congratulations, Eric.