(W0930-11-64) Developing an Assessment Framework, Items, and Expert Reviews for an Educational Instrument for the Proper Use of the USP/NF Pharmacopeia

Purpose: Pharmacopeias contains legally recognized standards for medicinal products. The proper use of these official compendia is an important trait for personnel in Pharma research and development (R&D) groups as well as quality control (QC) laboratory operations.  A survey was administered to 56 randomly selected QC-related personnel from Australia, Canada, Kenya, Nigeria, Tanzania, Uganda, and the USA. The results from 32 respondents, showed that 65% use United States Pharmacopeia/ National Formulary (USP/NF) as the primary Pharmacopeia, 25% use British Pharmacopeia, 6% use International Pharmacopeia, while the remaining 4% use other additional Pharmacopeias (Okezue 2022). Most information in the USP/NF is linked across similar performance tests and is not readily assessable using intuitive skills or regular academic training. To determine competencies, instruments (scales) are developed to create reliable measurements of underlying constructs. It is important that a scale measures the intended domain so that the results can serve the intended purpose (Raykov & Traynor, 2016). Some of the recommendations for developing a new psychometric instrument include stating the purpose of the scale, clearly defining the domain of the construct to be measured, determining if a measurement already exists, and other steps to validate the instrument (Bandalos, 2018). The current research covered the assessment framework, writing the items, and expert review for one instrument to be used to measure a key competency for Pharma R&D and QC personnel.

Methods: Searched for an already existing instrument in the Mental Measurement Yearbook (MMY) and Google Scholar to Investigate the existence of an already published instrument for the desired trait. Developed a video tutorial on how to properly use the USP/NF Pharmacopeia. Generated test items (questions) to assess content knowledge on the proper use of the USP/NF pharmacopeia. Developed an assessment framework for the proposed instrument; the domain for the construct was defined, delineating what the instrument measures. Defined the item domain, described the behavior measured by each test item, and proposed the cognitive process expected for each selected-response item. Provided a marking key and points for scoring the test items. To critique the educational instrument, the items were given to 2 subject matter experts (with > 10 years of experience as QC experts with the USP/NF) to independently review and provide feedback on the level of adequacy for each item. Following the approved protocols in Purdue IRB-2019-518, the proposed instrument was pre-tested using 3 cohorts. The Qualtrics test Items were administered as pre-, and post-tests following 1-hour teaching.  The percentage (%) correct responses for the pre-test for each cohort was compared to the % correct responses for the same items in the post-test. The difference in the group means was measured at a 95% confidence interval, α = 0.05. The t-test was used to evaluate the mean differences in the % correct responses. A significant p-value < 0.05, will reject the null hypothesis that the training did not impact the % of correct responses to the test items in the post-test evaluations.

Results: There were no existing equivalent instruments found in the MMY or Google Scholar databases relevant to the trait measured by this study. A list of definitions of the terms used is provided in Table 1. A video tutorial was created to provide the background information needed to enhance Pharma personnel competency. The test items developed for the instrument covered information related to the history of the USP/NF, the intervals for publication, regulatory applicability (enforcement) of the compendia, use of the General Chapter, General Notices, and individual Monographs of the compendia, and finally proper interpretation of technical terms. The two subject matter experts’ inter-rater reliability [IRR], which is the extent to which raters ranked order item adequacy in the same way, was calculated to be 0.8, see Table 2. After receiving the tutorial, there were significant differences in the means for % correct responses for cohort 1 [pre-test = 52%, post-test=76%, p-value = 0.0031], cohort 2 [pre-test = 55%, post-test=76%, p-value = 0.0023], and cohort 3 [pre-test = 50%, post-test=68%, p-value = 0.0036], Figures 1 and 2. The items listed in the USP/NF instrument can measure the behaviors exhibited in the trait under consideration. In interpreting the scores, a respondent should have a total score not lower than 90% to qualify as having an appreciable knowledge of how to effectively use the USP monograph. Any score less than 50% may be an indication that the respondent is at a beginner level, 50% to 75%, suggests intermediate proficiency. A total score of 75% to 90% is desirable, while 100% may indicate an expert level of cognitive knowledge on the proper use of the USP.

Conclusion: The p-values from the t-test results suggest that there was a significant improvement in the cohorts’ understanding of how to use the USP/NF after they received the training video developed in this study.

References: Okezue, Mercy A (2022): Developing Multipronged Models to Enhance Effectiveness of The Management System of Quality Control Laboratories. Additional Focus on Synthesis and Characterization For 5 New Salts of Bedaquiline. Purdue University Graduate School Thesis. https://doi.org/10.25394/PGS.19620642.v1
Raykov, T., & Traynor, A. (2016). Evaluation of multicomponent measuring instrument reliability in complex design studies. Structural Equation Modeling: A Multidisciplinary Journal, 23(2), 270-277.
Bandalos, D. L. (2018). Measurement theory and applications for the social sciences: New York, New York ; London, England: The Guilford Press.


Figure 1 Percentage of correct responses to individual items in the proposed USP/NF instrument


Figure 2 Significant outcomes were observed from the box plots, mean values, and t-test statistics for Pre-test versus Post-test assessment for all cohorts.


Table 1 The definitions for the terms used in developing the USP/NF instrument