Befriending Statistics 2: Update on the work of the Analytic Hub
It's a lot.... So, if you don't want to read the whole thing, I'll understand...
Analytic Hub Research Summary
Cheryl P. Talley, Ph.D.
November, 22, 2022
A major goal of the STEM-Us Center's Analytic Hub is to examine findings from among the STEM-US Center’s partners to better elucidate the comprehensive nature of effective and impactful STEM intervention strategies. Ultimately, the goal is to answer the question- “Why STEM Interventions work at HBCU’s". The answers to that question come from from many levels of analysis: focusing on the individual student, the classroom experience, the influence of the faculty or instructor, the activities of the department and the culture of the institution. That last one particularly, institutional culture, acknowledges the fact that while HBCU’s are a unique type of institution, they are by no means uniform in the type of learning environment that is offered to their respective students. Therefore answering the questions of “What works and why?” has required collaboration and partnerships that extend well beyond the idea of cooperatively working together on a particular project. The work of the Analytic Hub requires co-creative relationships that not only extend across disciplinary lines but also foster true community building among our colleagues. These relationships are created and supported in Communities of Practice.
During this third year of funding three Communities of Practice have emerged that together speak to creating and relating together in order to accomplish the shared goals of effective interventions and dissemination of the research efforts of HBCU faculty. The three Communities of Practice coalesce around Writing and Self-Care (CareFull Scholars Program); Teaching as Intervention (VSU Metacognition Group) and Data Analysis and Data Repository (Psychometrics Working Group). Each community uses both translational and transdisciplinary research approaches to disseminate information about HBCU students and faculty
During the first year of funding, we believed that a translational research approach was needed in order to examine the behavioral change that are an inherent part of any interventions. We understood that academic interventions are in essence psychological interventions. Therefore, theories and frameworks used by the Hub came from the social sciences. The Center began with the Phenomenological Variant of the Ecological Systems Theory by Margaret B. Spencer, (PVEST) as a theoretical framework. As a phenomenological theory, PVEST is one of few development theories that acknowledges the influence of race- the historical legacy of racism, contemporary forms of systemic racism and the relationship between race and power structures on minoritized communities. As an ecological theory, PVEST lends itself to examination of multiple levels of analysis. As a systems theory, PVEST can support longitudinal investigations across levels of analysis and across time. However, it soon became clear that our faculty partners, who were not trained in the social sciences, did not need to understand everything about PVEST or an entire field of psychological study to implement effective interventions. A transdisciplinary perspective was also called for so that the information flow could be bi-directional. We were learning from our partners as much as they were learning from us.
In the last year, there have been several iterations of the research methodology including the creation of new more specific partnerships. Analytic Hub Co-Director, Dr. Gail Hollowell along with STEM-US Principal Investigator Larry Blume, are creating a Community of Practice based on Course-Based Undergraduate Research Experiences (CURES). Analytic Hub Co-Director, Dr. Avis Jackson along with her colleague, Dr. Damon Brian at Morgan State seek to validate our STEM-US Assessment Instrument as we prepare it for broader use. Senior advisor to the Analytic Hub, Michelle Chatman is lead author on a proposed book chapter that will chronicle the new movement of "generative dissemination," the focus of our CareFull Scholars program. Finally, STEM-US Center investigator, Dr. Rihanna Mason is leading an effort to formulate another Community of Practice with Psychology faculty at 2-year institutions (Psy Beta). This new population of external partners represent faculty who, in many cases serve a similar demographic as HBCUs.
In addition to expanding our Communities of Practice, we have also adopted new statistical tools. The PVEST informed assessment instrument that was adopted by the Hub in year 2 was created at Virginia State University (Scherer, Talley, Fife, 2017). The data set associated with the 2021 STEM-US Assessment includes over 3000 incoming VSU STEM students. The assessment focuses on hypothesized protective and risk factors as experienced by the student. This data was first analyzed using stepwise linear and hierarchical regression in order to determine significant predictors of first semester GPA (Scherer, Talley & Fife, 2017). This was followed by analysis using principal component factor analysis, linear regression and PROCESS modeling (Davis 2022) to examine the mediating or moderating effects of the identified characteristics and to assess STEM course performance trends. The model identified the effective use of Academic Skills and Self Efficacy (an individual's belief in their own capacity to reach a particular goal) as directly related to GPA. We also found that the effect of Adverse Childhood Experiences has a moderating role in first semester GPA. (The role of negative childhood experience re-emerged in later analyses).
Additional Bayesian Analysis built upon the identification of latent factors and examined which test questions or items were together associated with high first semester GPA. The three that were identified from assessment items were academic skill-set, self-confidence and self-agency. Again, it was no surprise that self-confidence and academic skill were identified. The new finding was self-agency, which we operationalize as taking “ownership” of one’s education. These three constructs were positively associated with a high first semester GPA.
The most recent analysis used Psychometric Networks to examine the influence of protective and risk factors. The analysis indicated having a family member with a mental illness and also having experienced food insecurity were risk factors that were associated negatively with first semester GPA.
The inclusion of grade data is one unique characteristic of this data set and the data collection continues at VSU. We now have data from student cohorts that were attending VSU pre-Covid, during the transition to remote learning and when in-person classes resumed. This large and growing data set can be examined using various and more sophisticated statistical methods.Taken together, these data, suggest an emerging profile for a successful HBCU student and also areas which perhaps should be targeted for impactful interventions. The down side is that it only provides information about students from one HBCU campus. A future data repository is needed to collect similar data from many HBCU's. Without that, it will be difficult to elucidate a generalizable student profile.
The ultimate benefit of having a profile of a successful first semester student would be the possibility of tailoring academic interventions to a specific target audience and eventually, with the help of AI, making predictions based on certain personal factors According to our findings, so far, the findings obtained from the assessment are informing campus-wide activities at VSU. So we know that this model could work.
We now know that VSU entering students with high first semester GPA are more likely to:
envision their end goal and future self and see their current activities as contributing to the end goal
adopt, practice and maintain sound academic skills,
build self-confidence by identifying success at multiple points, not just final grades and also
take ownership of their education by assuming responsibility for their learning.
There should also be an acknowledgement that for students who have experienced childhood hardships the stress of the first semester may be enough to trigger coping responses that may negatively impact academic performance. At VSU, members of the Metacognitive Teaching group are particularly sensitive to these unique student needs and are willing to include evidence-based support in their classroom teaching. This new Community of Practice includes faculty from Chemistry, Mathematics, Biology, Political Science and even colleagues from Student Affairs.
In the coming year, the Hub hopes to build on the last 3 years of findings by examining if different types of interventions target (purposefully or inadvertently) these characteristics found with high performing VSU entering students. One such intervention, developed by STEM-Us Center investigator, Curg Muldrow is Scientific Literacy. Now under the direction of Drs. Brittany Chambers and Adrian Neely, Scientific Literacy is being piloted at Dilliard University, Elizabeth State as well as Morehouse College . It is hoped that data from the STEM-US Assessment instrument will someday be obtained from these students and other HBCU students, as well.
Here's another reason why collecting more data from other institutions is so important.
The current Hub studies do not lend themselves to examine aspects of the department or the culture of the institution. We also have no information about the student other than what was captured at the time the assessment was given. This represents a shortcoming in ecological validity and will further hamper our ability to generalize our findings. It is hoped that in the coming months new partnerships will emerge that will enable us to add to this historic and unprecedented work regarding HBCU students.
Information from this document is free to be shared but please cite this source: Talley, C.P. (2022) Analytic Hub Research Summary, November 2022 NSF # 2010676