Skip to main content
Download PDF
Download ePub

Organizational culture and climate profiles: relationships with fidelity to three evidence-based practices for autism in elementary schools

Implementation Science volume 14, Article number: 15 (2019) Cite this article

Abstract

Background

Implementation researchers have typically studied organizational culture and climate by testing whether individual dimensions are linked to the implementation of evidence-based practices (EBPs) rather than examining how the overarching social context influences implementation. This approach may limit implementation theory and strategy development to the extent that individual dimensions of culture and climate interact, mutually reinforce or counteract one another, or exhibit non-linear relationships. This study tests whether empirically identifiable culture and climate profiles emerge in a sample of organizations and examines how these profiles relate to EBP fidelity and work attitudes that support EBP sustainment, focusing on three EBPs for youth with autism delivered in schools as an example.

Methods

The study included 65 elementary schools in the U.S. that implemented three EBPs—discrete trial training, pivotal response training, and visual schedules—for youth with autism. Organizational culture and climate and work attitudes were assessed using the Organizational Social Context measure at the beginning of the school year. Observations of EBP fidelity occurred mid school-year. We used bias-adjusted stepwise latent profile modeling to (1) identify subpopulations of schools that share similar culture and climate profiles, and (2) test for mean differences across profiles in observed EBP fidelity and teacher and staff work attitudes.

Results

Controlling for region, four profiles best characterized the organizational cultures and climates of schools. Teachers and staff in schools with a comprehensive profile (high proficiency culture, positive climate) exhibited higher fidelity to two of three EBPs (d’s = .95 to 1.64) and reported superior work attitudes (d’s = .71 to 1.93) than teachers and staff in all other schools. Teachers and staff in supportive schools (low rigidity culture, positive climate) had better work attitudes, but not better fidelity, than those in schools with indifferent (low culture/climate, elevated stress) and constrained (high rigidity and resistance, high stress) profiles.

Conclusions

Organizational culture and climate profiles are a strong predictor of EBP fidelity and work attitudes that support EBP sustainment, highlighting the importance of an organization’s overarching social context when developing implementation theory and strategies. Strategies that foster a comprehensive profile may improve EBP implementation.

Peer Review reports

Background

Theories and frameworks that explain the implementation of evidence-based practice (EBP) in health care construe an organization’s social context, including its culture and climate, as a general organizational characteristic that affects the organization’s overall functioning and employees’ skillful and effective use of targeted EBPs [1,2,3,4]. Consistent with these theories and frameworks, research has linked several individual dimensions of culture and climate to the implementation of EBPs and to factors that support the sustainment of EBPs, such as staff job satisfaction, organizational commitment, and reduced turnover, across a wide range of healthcare settings (see [5] for a review).

This research has led to important advances in identifying targets for implementation strategies; however, this approach to studying culture and climate may underestimate the importance of the total social-psychological context within work environments [6]. One of the goals of implementation research is to understand how the overall social context of a work environment influences the implementation behavior of individuals within that environment [7, 8]. This goal may be compromised when holistic constructs such as culture and climate are broken down into individual dimensions that are measured and analyzed independently [9, 10]. For example, studies that include multiple dimensions of culture or climate in a single linear regression model control for the inter-relationships among these dimensions and implicitly assume that the effects of these variables are additive and independent. This assumption is incorrect if individual dimensions of culture and climate interact, mutually reinforce or counteract one another, or exhibit non-linear relationships such that the overall social context is not equal to the sum of its parts [6]. Recent studies have highlighted the fact that different dimensions of organizational social context interact in complex ways to affect providers’ implementation behaviors [11], underscoring the need for research that takes a nuanced and holistic approach to the study of culture, climate, and EBP implementation. The goal of this study is to examine how overarching configurations or profiles of organizational culture and climate relate to fidelity of EBP implementation and to work attitudes that support the sustainment of EBPs.

In a profile approach, investigators empirically identify subpopulations of organizations that share similar response patterns across an interrelated set of variables and examine how these subpopulations differ on outcomes of interest [12]. For example, some organizations might have a culture and climate profile that is characterized by lower than average scores on all dimensions, whereas other organizations might have a profile that is high on some dimensions and low on others. These differing profiles reflect the nuanced ways in which individual dimensions of culture and climate move together as they reinforce, attenuate, or otherwise modify each other’s effects. By observing the types of culture and climate profiles that emerge in healthcare organizations and testing differences in how these profiles relate to implementation fidelity and factors that support EBP sustainment (e.g., positive work attitudes and reduced turnover), investigators can gain insight into how an organization’s overarching social context relates to EBP implementation, with implications for developing implementation theory and strategies.

This study focuses on the implementation of three well-established EBPs for youth with autism delivered in public elementary schools in the U.S. Little research has examined the relationships between culture and climate and fidelity to EBPs in schools [13, 14], despite the fact that schools comprise the single largest provider of behavioral health services for youth in the U.S. [15, 16]. Although schools are increasingly expected to implement EBPs [17, 18], researchers have consistently found that EBPs often are not implemented in schools and when they are, fidelity of implementation is typically poor [19,20,21,22]. School-based EBPs are especially important for youth with autism, as 52% of these youth receive mental health treatment in school [23]. Identifying factors, such as culture and climate profiles, that are associated with high fidelity to EBPs for youth with autism in schools has strong potential to improve the well-being of this population and to advance implementation theory.

We draw on Glisson and colleagues’ [24] empirically supported theoretical model of organizational culture and climate which was developed specifically to characterize the social contexts of child-serving organizations that deliver behavioral health services. Consistent with the broader literature, this model posits that organizations are characterized by different types of cultures and climates that contribute to differential effectiveness [25,26,27]. Glisson’s model defines organizational culture as the shared norms and behavioral expectations that guide how work is prioritized and completed in the organization and includes three culture dimensions: proficiency, rigidity, and resistance [24]. Proficiency refers to shared staff perceptions that they are expected to be responsive to client needs and client well-being as their top priority and to maintain competence in up-to-date treatment models. Rigidity describes staff perceptions regarding their autonomy in completing work tasks versus the requirement to closely follow prescribed rules. Resistance describes staff perceptions regarding the extent to which they should actively or passively resist new ways of working and maintain the status quo.

The model defines organizational climate as staffs’ shared perceptions of the impact of the work environment on their personal well-being [28]. The three dimensions of climate in this model are functionality, engagement, and stress [24]. Functionality refers to staff perceptions that they receive support and cooperation from peers and supervisors, clearly understand their role, and have opportunities for growth and advancement. Engagement describes the extent to which staff remain personally involved in their work and accomplish meaningful outcomes. Stress captures shared perceptions of the extent to which role conflict, role overload, and emotional exhaustion interfere with staffs’ ability to do a good job.

Several observational and experimental studies from service settings for youth (e.g., specialty mental health clinics, child welfare agencies) have linked these individual dimensions of culture and climate to superior implementation of EBPs [29,30,31,32,33,34] and to other outcomes such as service effectiveness and employee work attitudes [35,36,37]. However, no studies have empirically assessed the overarching culture and climate profiles that may be present in organizations that deliver EBPs nor tested the relationships between these empirically-identified profiles and EBP fidelity. These are important gaps for two reasons. First, implementation theory that describes how multiple dimensions of organizational culture and climate simultaneously influence EBP use has yet to be articulated and thoroughly tested. Many studies assume (either implicitly or explicitly) that culture and climate dimensions have independent and additive effects; these assumptions are embedded within the hypotheses and statistical models. To the extent that these assumptions are incorrect, the models testing them are misspecified and the field’s understanding of social context as a mechanism for supporting EBP implementation and sustainment is at best incomplete and at worst distorted. Investigators may erroneously conclude that organizational social context does not play a role in implementation or that only certain dimensions serve as mechanisms to support implementation when in fact it is a pattern of dimensions (or potentially multiple patterns of dimensions) that together create the necessary and sufficient conditions for successful EBP implementation. Second, these deficits in theory stifle the development of optimally effective and efficient implementation strategies. Drawing on a faulty theory, investigators may develop strategies that are ineffective or they may fail to pursue strategies that might prove more effective or efficient than alternative approaches.

Study aims

First, we tested the extent to which an interpretable set of organizational culture and climate profiles emerged in a large sample of elementary schools that deliver EBPs to youth with autism. Our primary goal was to empirically identify subpopulations of organizations that shared similar culture and climate profiles. Second, we tested whether these empirically-derived culture and climate profiles were related to two types of outcomes: (1) observed teacher and staff fidelity to three EBPs for youth with autism and (2) teacher and staff job satisfaction and organizational commitment, which comprise work attitudes that are linked to reduced turnover and have been hypothesized as antecedents to improved EBP sustainment [38,39,40]. Examining both types of outcomes is important because some profiles may support multiple types of EBP-related outcomes (e.g., fidelity, sustainment) whereas other profiles support only a single type of EBP-related outcome.

Method

Participants and setting

We conducted an observational study of factors related to fidelity to three EBPs for autism in 65 schools that contained kindergarten to third-grade special education classrooms for youth with autism and that had elected to implement behavioral health EBPs for youth with autism as part of their standard curricula (see [41]). During the last 20 years, a growing number of school districts across the USA have sought to better meet the needs of students with autism by providing training and coaching to their teachers and staff in the use of EBPs based on principles of applied behavior analysis [42]. In partnership with school district officials presiding over districts in the northeastern and northwestern USA that had elected to implement EBPs, our team designed a study to identify factors related to high and low levels of fidelity in these schools in order to generate targets for implementation strategies.

Ninety-two schools were invited to participate in the study because they had a kindergarten-third grade special education classroom with students with autism and planned to provide training and coaching to their staff to implement the target EBPs. Of these 92 schools, 18 declined to participate, 7 had fewer than three staff working in their autism support classroom or provided substantially missing data (i.e., > 30%) on study measures that prevented data aggregation, and 2 were multivariate outliers on study measures, resulting in a final analytic sample of 65 schools (71%), which included 86 kindergarten-through-third grade autism support teachers, 161 classroom staff, and 58 principals serving children with autism. Four schools had three participating autism support classrooms; 13 schools had two participating autism support classrooms, and all other schools had one autism support classroom. Table 1 presents participants’ demographic characteristics.

Table 1 Characteristics of study participants
Full size table

Evidence-based practices for autism

School districts contracted with a purveyor organization to train all teachers and staff who worked in autism support classrooms in three closely related EBPs which are based on principles of applied behavior analysis (see [41]): discrete trial training (DTT), pivotal response training (PRT), and visual schedules (VS). DTT is a highly structured, one-on-one instructional method that uses massed trials and instructional cues from the teacher/classroom staff to elicit a targeted response from the student with autism [42,43,44]. PRT is a naturalistic, play-based behavioral intervention that leverages “teachable moments” to train students with autism in generalizable language and play skills (e.g., responsivity to the environment) [45]. VS is a classroom-wide intervention that provides visual cues for students with autism to transition from one activity to another [46]. All three EBPs share a theoretical basis in applied behavior analysis and have strong empirical support [42,43,44,45,46]; consequently, school officials and the purveyor organization presented the EBPs as an integrated set of tools that teachers were expected to use to meet the needs of students with autism related to language, social skills, and pre-academic skills. PRT and DTT are delivered as one-on-one interventions in sessions that typically last approximately 15 min with a student in the classroom [20; 41]. VS is implemented classroom-wide and includes visual prompts that are posted in the classroom or used throughout the day, as well as modifications to how the schedule and transitions are managed [46].

EBP training and coaching

Training and coaching for the three EBPs occurred via an integrated process. The initial training session occurred prior to the start of the school year and included a two-day interactive learning session as well as distribution of written materials including a manual detailing all three interventions and requisite materials. The initial two-day training focused on the theory and rationale for the interventions, their evidence base, steps in using them in the classroom, and strategies for implementing the interventions via a team approach. Teaching strategies included didactics, video examples, in vivo modeling, role-play, discussion, and “question and answer” sessions focused on application to participants’ classrooms. Teachers were given time to create materials for the visual schedules intervention for use in their classrooms (e.g., individual and classroom schedules). In addition, teachers and staff were provided with lesson plans, toys, flash cards, and data sheets necessary to implement DTT and PRT and they practiced using these materials during the training.

Following the initial training, teachers and staff received monthly coaching in their classrooms related to all three EBPs. Coaching occurred in vivo during class sessions once per month and each session lasted approximately 2 h. Coaches observed teachers’ use of all three EBPs and provided didactic training, modeling, and feedback as well as support with strategies for managing behaviors so that the EBPs could be used to address target behaviors. Additional file 1 presents a Standards for Reporting Implementation Studies (StaRI) checklist with details regarding the implementation strategy used by the schools [47].

Procedure

The University of Pennsylvania and the University of Washington institutional review boards and the school districts provided ethics approval for the study. Recruitment for the study began by meeting with officials in school districts that had decided to implement EBPs for youth with autism to describe the study, acquire their permission, and obtain a list of elementary schools with self-contained classrooms for children with autism where EBP training would be provided. Next, we contacted the principal at each prospective school to set up an initial meeting and obtain their consent for the school to participate in the study. Following approval by the principal, we met with teachers and school staff who worked in the school’s autism support classroom to obtain their consent to participate in the study. Even if principals agreed to their schools’ participation, teachers and staff could decline survey completion. Training in the EBPs was required by the school districts and was provided to all autism support teachers and staff regardless of their participation in the study and completion of study measures. No modifications to the training or coaching were made as part of the study.

Following recruitment of participants, the study included two phases. First, participants at each school (i.e., principals, autism support teachers, and classroom staff) completed study measures (i.e., organizational culture, climate, and work attitudes) on-site at the beginning of the school year following the EBP training (November–December 2015). Survey completion took approximately 45–60 min and participants were compensated $50 USD for their time. Second, beginning one month after administration of the study measures (i.e., January 2016), a member of the research team visited each school and conducted two classroom observations approximately two months apart to assess fidelity to each EBP in the school’s autism support classrooms. Observations were scheduled during periods of the school day when teachers/staff informed the research team they would be using all three of the EBPs. Observations were conducted in the middle of the school year (January to April) to ensure that teachers and staff had time to learn and implement the EBPs. Data collection was avoided during the beginning (prior to survey administration) and end of the school year when EBP implementation slows.

Measures

Organizational culture and climate

Organizational culture and climate were measured using the Organizational Social Context (OSC) measure, a 105-item instrument that assesses organizational culture, climate, and work attitudes in child-serving settings [24]. Scores on the OSC have demonstrated reliability and validity in several studies [30, 33, 37, 48]. In partnership with the OSC developers, the research team made minor adaptations to the wording of items (e.g., referring to “students” instead of “clients” and “teachers and classroom staff” instead of “members of my organizational unit”) to ensure its use was appropriate among teachers and staff in the school context [41]. Items ask respondents to indicate the extent to which each statement characterizes their workplace on a scale from 1 (Never) to 5 (Always). Statements refer to shared norms and expectations within the organization (i.e., culture) and the impact of the work environment on the individual’s personal well-being (i.e., climate). Administration takes approximately 20 min.

The OSC provides scores on six subscales reflecting the dimensions of culture (i.e., proficiency, rigidity, resistance) and climate (i.e., engagement, functionality, stress) described above. Subscale scores are generated by aggregating (i.e., averaging) participants’ individual responses to the school-level after evidence of within-group agreement and between-group variance has been provided to support the construct validity of the aggregated, school-level scores (see [49] for details). Aggregation is based on the underlying theory of culture and climate, which posits that these constructs are organization-level variables, not characteristics of individuals [50]. Evidence of staff inter-rater agreement on each OSC scale (i.e., proficiency, rigidity, resistance, engagement, functionality, stress) was provided by the rwg(j) statistic [51, 52] where values >  .7 are considered acceptable [53]. In this sample, inter-rater agreement was excellent on all six subscales (mean rwg(j) = .92, range = .88 to .97), indicating respondents exhibited high levels of agreement in their perceptions of culture and climate.

The OSC also measures individual-level work attitudes with two scales assessing the individual’s job satisfaction and organizational commitment [54]. Job satisfaction represents a positive appraisal of one’s job tasks and work assignments, whereas organizational commitment represents a willingness to exert considerable effort on behalf of the organization and a strong desire to remain a member of the organization [55]. In prior studies, these scales have demonstrated excellent reliabilities and evidence of validity [24, 54]. We aggregated individual job satisfaction and organizational commitment scores to the school level to reflect the mean level of these constructs in the schools.

EBP Fidelity

Consistent with best practices [19, 20, 56], a research assistant rated DTT, PRT, and VS fidelity via direct observation in each classroom using an EBP-specific fidelity checklist. The checklists were developed in previous studies examining fidelity to these practices in a classroom setting; in prior research, scores on the checklists demonstrated convergent validity with (1) improvement in youth outcomes and (2) self-reported teacher fidelity (see [19, 20, 56]). The fidelity checklists included 11 items for DTT (α = .97), 17 items for PRT (α = .97), and 10 items for VS (α = .93). Items on each checklist assessed the extent to which teachers and staff implemented the core components of the EBP during a 10–15-min observation session using a Likert scale from “0” (does not implement) to “4” (highly accurate implementation). Because teachers and staff within the schools worked as integrated teams to meet the needs of all students within their classrooms, fidelity assessments did not focus on specific teachers or staff per se, but rather on the fidelity with which EBPs were implemented by team members observed during the observation period. Observations of each EBP were limited to 15 min, as this was the allotted time for teachers and staff to work one-on-one with each student with autism to ensure all students in the classroom had the opportunity for one-on-one intervention. VS implementation generally occurred throughout the observation period when students with autism transitioned from one activity to another. For each practice, the items were averaged to calculate a total score for the observation period. Consistent with our conceptualization of teachers and staff as teams that meet the needs of a target student population, fidelity scores for each EBP were averaged across the two observations to yield an average score for each of the three EBPs in each classroom. In schools with more than one autism support classroom, fidelity scores were subsequently aggregated to the school level for use as a school-level outcome variable. Research assistants were trained to 90% reliability on each fidelity measure through didactic instruction and coding of training videos prior to conducting field observations [57].

Data analysis

We used bias-adjusted stepwise latent profile modeling to identify subpopulations of schools that shared similar culture and climate profiles (conditioned on region) and test for mean differences across profiles on observed EBP fidelity and teacher and staff work attitudes that support EBP sustainment [58,59,60,61]. The first step identified subpopulations or classes of schools that shared similar organizational culture and climate profiles based on their organization-level scores on the six subscales of the OSC (i.e., proficiency, rigidity, resistance, engagement, functionality, stress). Scores on each dimension of culture and climate were standardized prior to analysis and all analyses included region (i.e., northwestern vs. northeastern USA) as a covariate to control for differences in resource constraints, policy environments, and student populations across regions. Consistent with best practices, the best fitting model was selected on the basis of: model fit criteria (i.e., lower values on the sample size-adjusted Bayesian information criterion indicate better fit; [62, 63]); entropy (i.e., values closer to 1 indicate greater separation between classes; [64]); sufficiently populated classes (i.e., no less than 5% of the sample in a given class); probabilities of correct classification (i.e., values above .8 are typically considered adequate [65]); and, interpretability of classes based on alignment with previous research and theoretical considerations [65, 66]. In the analysis, the categorical latent variable represents a set of homogeneous subpopulations of schools that share similar organizational culture and climate profiles on the OSC, conditioned on region [61].

The second step tested mean differences across classes on the distal outcomes of EBP fidelity and staff work attitudes using a weighted ANOVA which accounts for the probabilistic uncertainty of classification, as described by Bolck et al. [59], and recommended by methodologists (i.e., the Bolck–Croon–Hagenaars (BCH) approach [60]). This procedure incorporates robust standard errors and uses a Wald test to provide an omnibus test and pairwise comparisons of the equality of means on the distal outcomes across the latent classes. Simulation research indicates the BCH approach yields unbiased estimates and robust tests of significance even when statistical assumptions are violated [58,59,60, 67]. All analyses were implemented in Mplus, Version 8 via the TYPE = MIXTURE analysis command with the BCH specifier in the AUXILIARY variable command [68]. Effect sizes were calculated using Cohen’s d which expresses the distance between means in standard deviation units of the dependent variable [69].

Results

School culture and climate profiles

Results of the latent profile analysis (LPA) indicated a four-class solution best characterized the culture and climate profiles of the schools in this sample, as indicated by model selection criteria (i.e., lowest sample size-adjusted Bayesian information criterion (SSA-BIC) value), good separation between classes (entropy = .87), high average latent classification probabilities (range = .86 to .99), adequate populations in each class (i.e., no classes with < 5% of the sample), and theoretically interpretable culture and climate profiles. Figure 1 shows the culture and climate profiles of the four classes derived from the analysis.

Fig. 1
figure 1

Organizational culture and climate profiles in schools. Note: Organizational culture and climate profiles are based on bias-adjusted stepwise latent profile analysis incorporating the six Organizational Social Context subscale scores and controlling for region. N = 65 schools

Full size image

The first profile included 35% of schools (n = 23) and was labeled indifferent. This profile exhibited low scores on every dimension of culture and climate except stress, which was significantly higher than average (z = .68, SE = .22, p = .002) and .91 standard deviations higher than any other dimension of culture or climate in the profile. Levels of proficient culture (z = − .77, SE = .17, p < .001), engaged climate (z = − .73, SE = .23, p = .001), and functional climate (z = −.76, SE = .16, p = .000) were lower in this profile than in any other profile, suggesting teachers and staff in these schools do not experience norms and expectations to improve student well-being or remain competent in up-to-date practices, nor do they experience the cooperation and support they need from colleagues to perform their jobs. Levels of rigid culture also were significantly lower than average in this profile (z = − .46, SE = .16, p = .004), indicating teachers and staff perceive minimal oversight or direction from leadership and wide latitude to complete their job tasks.

The second profile included 20% of schools (n = 13) and was labeled constrained. Schools with a constrained profile were characterized by the highest levels of rigid culture (z = 1.37, SE = .25, p < .001) and resistant culture (z = .81, SE = .28, p = .004) observed in the sample, exceeding all other profiles by > 1 and .67 standard deviations, respectively. These scores also exceeded the level of proficient culture in this profile (z = .52, SE = .25, p = .042). Teachers and staff in these schools report minimal levels of autonomy and strong normative pressure and expectations to reject new ways of working. The most elevated dimension of climate was stress, which was significantly higher than average (z = .67, SE = .17, p < .001).

The third profile included 35% of schools (n = 23) and was labeled supportive. These schools had average scores on proficient culture (z = .03, SE = .17, p = .858) and resistance (B = − .33, SE = .20, p = .102) and significantly lower than average scores on rigidity (z = −.46, SE = .14, p = .001), suggesting norms and expectations that support teacher autonomy but are not above average with regard to maintaining competence or adopting new innovations. The most prominent feature of this profile was the pattern of climate scores in which stress was nearly a standard deviation below average (z = − .91, SE = .17, p < .001) and both functionality (z = .59, SE = .21, p = .006) and engagement (z = .34, SE = .19, p = .075) were more than a standard deviation higher than stress. Teachers and staff in these schools share perceptions that their work environment supports their personal well-being and provides relatively high levels of autonomy even as expectations for maintaining competence and incorporating innovations into practice do not exceed average.

The fourth culture and climate profile included 9% of schools (n = 6) and was labeled comprehensive. The level of proficient culture in this profile (z = 1.63, SE = .20, p < .001) exceeded any other profile’s level of proficient culture by > 1 standard deviation. This was accompanied by the highest levels of engaged climate (z = 1.49, SE = .18, p < .001) and functional climate (z = 1.25, SE = .46, p = .006) observed in the sample. In addition, these schools were characterized by average levels of rigid culture (z = .23, SE = .33, p = .706) and resistant culture (z = .13, SE = .42, p = .754) and a slightly lower than average level of stress (z = − .58, SE = .44, p = .193), reflecting staff perceptions of typical expectations related to the hierarchy of authority (i.e., rigidity) and approach to innovations (i.e., resistance) in a school setting. Teachers and staff in this profile report that they are expected to focus on student well-being and maintain competence in up-to-date practices and they perceive the support and cooperation they need from their colleagues and supervisors to do a good job.

Relationships between school culture and climate profiles and EBP Fidelity

Table 2 presents the results of the bias-corrected omnibus tests and pairwise comparisons examining mean differences in EBP fidelity across the four culture and climate profiles on each EBP. Teachers and staff in comprehensive schools exhibited significantly higher fidelity to DTT than teachers and staff in supportive (d = 1.01, p = .010), indifferent (d = .95, p = .009), and constrained schools (d = 1.64, p < .001); however, there were no differences in DTT fidelity among teachers and staff in supportive, indifferent, and constrained schools. Similarly, teachers and staff in comprehensive schools exhibited higher fidelity to PRT than teachers and staff in supportive (d = 1.21, p = .001), indifferent (d = 1.21, p < .001), and constrained schools (d = 1.42, p = .001), even as the level of fidelity to PRT in these three school profiles did not differ from each other. There were no significant differences on fidelity to VS across the culture and climate profiles (p = .728).

Table 2 Differences in fidelity to evidence-based practices and teacher work attitudes by culture and climate profiles
Full size table

Relationships between school culture and climate profiles and collective teacher and staff work attitudes

Results of the bias-corrected omnibus tests and pairwise comparisons examining mean differences on teacher job satisfaction and organizational commitment across the four culture and climate profiles are shown in Table 2. Teachers and staff in schools with comprehensive profiles reported significantly higher job satisfaction than teachers and staff in supportive (d = .71, p = .033), indifferent (d = 1.61, p < .001), and constrained (d = 1.93, p < .001) schools. In addition, teachers and staff in supportive schools reported significantly higher job satisfaction than teachers and staff in indifferent (d = .90, p = .003) and constrained schools (d = 1.23, p < .001).

A similar but not identical pattern emerged for organizational commitment. Teachers and staff in comprehensive and supportive schools reported significantly higher levels of organizational commitment than teachers and staff in constrained and indifferent schools even though the levels of organizational commitment did not differ between the top two profiles (comprehensive vs. supportive, p = .058) or the bottom two profiles (constrained vs. indifferent, p = .493).

Post hoc analyses

The analyses presented above confirmed that different culture and climate profiles in schools are related to different levels of EBP fidelity and teacher and staff work attitudes, controlling for differences in resources, policies, and populations that characterize the two regions in this study. They do not, however, establish the factors that serve as antecedents to culture and climate profiles or clarify whether some third variable, such as financial strain, relates to both culture and climate profiles and EBP fidelity. While full examination of this issue is beyond the scope of this study, we partially addressed this by testing whether within-region variation in school size and the percent of students who received free or reduced lunch differed across culture and climate profiles. We conceptualized these variables as indicators of financial strain on schools and tested them using the bias-corrected BCH procedure described above.

Results of these analyses are presented in Table 3. Neither variable was supported as an antecedent to, or potential confound of, the relationship between culture and climate profiles and EBP fidelity or work attitudes. Schools with comprehensive profiles did not differ from any other profile on size (measured as the total number of students); although schools with indifferent profiles were significantly smaller than supportive schools (p = .041). Schools with comprehensive (96.15%) and constrained (96.88%) profiles did not differ from each other on the percentage of students who received free or reduced lunch (p = .921); however, these schools had significantly higher percentages of students receiving lunch assistance than schools with indifferent (69.87%) and supportive (69.58%) profiles. These analyses indicate that the positive fidelity outcomes observed in schools with comprehensive profiles are not related to school size or the percent of students receiving free or reduced lunch.

Table 3 Differences in school characteristics by school culture and climate profiles
Full size table

Discussion

This study extends implementation theory and research by using a profile approach to understand how configurations of organizational culture and climate dimensions relate to staff fidelity to three EBPs for youth with autism in public elementary schools [1,2,3]. Whereas prior studies have investigated the relative importance of individual dimensions of culture and climate for EBP implementation (e.g., [29, 70]), this study sought to understand how the overarching social context of culture and climate relates to EBP implementation outcomes [6, 12]. Results confirm that schools exhibited distinguishable organizational culture and climate profiles that predicted large differences in observed teacher and staff fidelity to EBPs as well as variation in teacher and staff work attitudes that support EBP sustainment. These findings extend prior research by showing that service delivery organizations exhibit distinctive culture and climate profiles—which reflect the unique ways that individual dimensions of culture and climate interact, reinforce, and counteract one another—and that these profiles are strongly related to EBP fidelity and factors that support EBP sustainment.

Results from this study have important theoretical implications and highlight areas for future research. First, no single dimension of culture or climate predicted EBP fidelity on its own nor did the linear combination of all six dimensions predict fidelity in an OLS regression model (see Additional file 2 for a bivariate correlation matrix and regression results). Instead, high fidelity was observed only when a comprehensive configuration of highly proficient culture, average levels of rigidity and resistance, and a highly positive organizational climate were all present. This suggests that dimensions of organizational culture and climate interact with, reinforce, and counteract one another in complex, non-linear ways as they relate to EBP implementation such that the overall gestalt of the social context may be more important than the level of a single dimension [11]. Research examining organizational social context profiles in other industries (e.g., banking, food distribution) suggests that the comprehensive and supportive profiles observed in this study may generalize to other types of organizations and may reflect the organization’s balance between a focus on strategic outcomes (e.g., fidelity) and employee well-being (e.g., climate) [6]. Clearly, there is a need for theory development and testing to better understand the types of culture and climate profiles that emerge in different types of organizations that deliver healthcare services (e.g., schools, hospitals, outpatient clinics), the antecedents to these different profiles, and how these profiles relate to implementation outcomes. Ostroff and Schulte [71] provide a review of methods for studying organizational culture and climate profiles, including the use of qualitative comparative analysis and other fuzzy set methods which focus on identifying the necessary and sufficient conditions for achieving defined outcomes [72].

Second, there is evidence that different profile patterns support different types of outcomes. Supportive profiles were related to superior teacher and staff work attitudes, which is consistent with the support these workplaces provide for staff well-being accompanied by the relatively high levels of staff autonomy. However, supportive profiles were not associated with superior staff fidelity to EBPs, possibly because of the lack of proficient cultural expectations to direct staff toward EBPs as an important part of effective service delivery (i.e., competence in up-to-date treatment practices). In contrast, teachers and staff working in schools with comprehensive profiles, which engendered both highly proficient cultures (along with average rigidity and resistance) and highly positive climates, had superior EBP fidelity and employee work attitudes. This is consistent with other studies showing an interaction between general organizational climate and more focused dimensions of organizational social context that support targeted behaviors [6, 11, 73]. Based on these results, we can speculate that different types of culture and climate profiles may support different types of implementation outcomes. Furthermore, these results advance causal theory in implementation science by suggesting that a comprehensive profile is one mechanism through which schools may generate optimal EBP fidelity and sustainment among their staff.

While caution is warranted in interpreting the results of a single study, we note some potential theoretical implications associated with the specific shape of the comprehensive profile. First, the extremely high level of proficient culture suggests that EBP fidelity may be optimized when an organization’s culture engenders a dual focus on improving student well-being and optimizing staff competence. It is important to note that items on the proficiency scale do not refer specifically to EBPs but rather to expectations for teacher competence in effective practices along with a focus on promoting student well-being. This contrasts with other more targeted organizational constructs, such as strategic implementation climate, which assesses the extent to which use of a particular EBP is expected, supported, and rewarded by the organization [74]. Thus, these constructs differ in their specificity and in the extent to which EBP implementation is viewed as a means to an end (i.e., proficiency) versus an end in itself (i.e., implementation climate). It may be that linking EBP implementation to higher-order goals that are valued by providers (e.g., improved client well-being) is helpful (or necessary) to activate the type of practitioner motivation that is necessary to achieve high fidelity to complex EBPs in real-world settings [75]. Conversely, proficient culture may serve as a catalyst for the development of a targeted implementation climate [76]. Second, comprehensive profiles pair a proficient culture with a positive climate, which indicates that staff within these schools perceive that their well-being is supported at work. This pairing may be essential for achieving optimal EBP fidelity because it ensures that providers have the requisite psychological health and well-being necessary to effectively learn and master new practices. Third, the comprehensive profile included average levels of rigidity and resistance as opposed to very low levels of rigidity and resistance. This suggests that some minimum level of rigidity and resistance within a culture may support the achievement of EBP fidelity by providing the requisite structure and healthy skepticism toward competing innovations that are essential to persevering with a new EBP long enough to achieve high fidelity. Additional research is needed to explore these hypotheses and to better understand how the more general organizational characteristics captured by the OSC relate to targeted dimensions of social context such as implementation climate, which are presumed to be most proximal to implementation outcomes.

Third, these results suggest there is not necessarily a linear relationship between organizational culture and climate as has been suggested by some theorists [26, 77]. For example, low levels of rigidity (culture) were associated with both high and low levels of stress (climate), suggesting that the experience of employee autonomy is not universally positive in schools. Similarly, profiles that were high on proficiency (culture) exhibited both high and low levels of stress (climate) depending in part on the accompanying levels of rigidity and resistance (culture). These nuances suggest that while certain dimensions of culture may be more strongly predictive of positive or negative climates in general [37], there are important exceptions in which the level of one dimension of culture may change the effect of another dimension. This is a ripe area for theory development.

Although additional research is needed, results from this study have implications for the implementation of EBPs for youth with autism in elementary schools and the development of strategies to support EBP implementation more broadly. First, these findings suggest that the development of a comprehensive organizational culture and climate profile may serve as a mechanism to support the delivery of EBPs (e.g., DTT and PRT) with high fidelity. Only schools with comprehensive profiles had significantly higher levels of observed fidelity to DTT and PRT and the magnitudes of these relationships were large (d’s = .95 to 1.64) suggesting they are practically important. Second, the positive relationship between comprehensive profiles and teacher and staff work attitudes suggests that this type of social context also may support EBP sustainment. Employee retention is a critical factor in maintaining high levels of EBP fidelity [38, 78] and work attitudes are a robust predictor of turnover [39, 40, 79]. To the extent that schools can foster comprehensive culture and climate profiles it is possible that they may be able to support EBP fidelity and sustainment by retaining teachers and staff.

We note that the relationship between OSC profiles and VS implementation was not significant. While there may be many reasons for this, we posit that organizational social context may not be as critical for VS implementation as it is for DTT and PRT. VS implementation is simpler in comparison to DTT and PRT and relies heavily on individual teachers and staff for the preparation of materials (e.g., laminated pictures) and organization (e.g., setting and resetting the schedules), whereas DTT and PRT are more resource intense in terms of staff time (e.g., 10–15 min one-on-one per student) and physical materials (e.g., toys, data sheets), which may require more collaboration and support from other members in the school.

Research in other healthcare settings (e.g., community mental health clinics) indicates that organizational culture and climate are malleable and can be changed with organizational interventions in as little as 18 months [32, 36]. These studies also suggest that improvement in targeted dimensions of organizational culture support improved EBP implementation [31, 34]. However, very little is known about the most efficient or effective ways to change organizational social context or the strategies needed to target specific dimensions. For example, different organizations may need to change their contexts in different ways (e.g., reduce rigidity versus increase proficiency) and consequently, the most effective implementation strategy may vary depending on the organization’s culture and climate profile.

A related area for future research involves identifying predictors of profile membership and understanding the factors that serve as antecedents to the development of comprehensive culture and climate profiles. Results of our post hoc analyses suggest that differences in school size and socioeconomic status of students do not explain why some schools have comprehensive profiles and other schools in the same region have constrained profiles. Identifying factors such as system- and organization-level leadership behaviors and organizational practices that predict profile membership is an important area for future research [4, 80, 81].

Limitations

It is important to interpret these findings within the context of their limitations. First, we did not conduct a randomized experiment, so the relationships in this study cannot be interpreted causally. Although we established the temporal precedence of organizational culture and climate profiles relative to teacher and staff EBP fidelity, we cannot say that the profiles caused EBP fidelity. It could be that higher EBP fidelity caused different patterns of culture and climate, although this seems unlikely because all teachers and staff received EBP training and support from the same source in a standardized way. More plausibly, a third variable may explain patterns of culture and climate and staff EBP fidelity. For example, the Consolidated Framework for Implementation Research [1] describes several factors (e.g., teacher characteristics, competing initiatives within the school environment) that may confound the relationship between culture and climate profiles and fidelity. This may occur as a confounding effect or as an indirect effect (e.g., leadership causes culture/climate which causes EBP fidelity and staff retention). We attempted to address this in part by controlling for all the policy, resource, and student population differences associated with geographic region in our analysis and by conducting post hoc tests with variables that have theoretical links to culture and climate (i.e., school size and percent of students with free/reduced lunch); however, a more definitive answer awaits experimental research. We are aware of one experiment in behavioral health clinics which showed that planned improvement in a single dimension of culture was associated with increased clinician adoption of EBPs [34]; however, experiments testing the holistic profiles described here have not yet been conducted.

Other limitations include the limited geographic distribution of our sample across northwest and northeast regions in the US. Studies that incorporate nationally representative samples of schools and samples in other countries are needed to replicate and extend our findings. In addition, the use of self-report measures to assess teacher and staff perceptions of organizational culture and climate is sometimes viewed as a weakness; however, we view this as a strength and essential feature of the study because staff respond to their shared perceptions of the social environment rather than the “actual” environment as might be reported by an outside observer [24]. In contrast, fidelity to EBPs is best measured through observational assessments because third-party raters can reliably and validly assess the completion of specified procedures and thereby circumvent self-recall and self-report biases. This study incorporated observer-rated fidelity to three different EBPs and linked these to staffs’ prior, shared perceptions of the social context, providing strong evidence for a link between culture and climate profiles and EBP fidelity. Finally, this study would have been strengthened with qualitative data and member checking around the findings. This is an important area that warrants additional research.

Conclusion

Results of this study highlight the importance of an organization’s overarching social context and the complex interplay among individual dimensions of organizational culture and climate as they relate to observed fidelity to EBPs and staff work attitudes that support EBP sustainment. The study identified four unique culture and climate profiles that characterized schools (indifferent, constrained, supportive, and comprehensive) and confirmed EBP fidelity was significantly superior in 9% of schools that exhibited a comprehensive profile. Furthermore, teacher and staff work attitudes that support the sustainment of EBPs were also superior in these schools. Examination of the comprehensive profile shape suggests that effective EBP implementation is a function of both a cultural focus on responsiveness to student needs and teacher competence as well as a supportive work environment. These findings highlight important directions for theory development in implementation science and suggest that implementation strategies should cultivate a comprehensive organizational culture and climate profile in order to improve EBP use for youth with autism.

Abbreviations

DTT:

Discrete trial training

EBP:

Evidence-based practice

LPA:

Latent profile analysis

OSC:

Organizational Social Context

PRT:

Pivotal response training

SSA-BIC:

Sample size-adjusted Bayesian information criterion

VS:

Visual schedules

References

  1. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012;43:337–50.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, Blachman M, Dunville R, Saul J. Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. Am J Community Psychol. 2008;41:171–81.

    Article  PubMed  Google Scholar 

  4. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health and Ment Health Serv Res. 2011;38:4–23.

    Article  Google Scholar 

  5. Williams NJ, Beidas RS. Annual Research Review: The state of implementation science in child psychology and psychiatry: a review and suggestions to advance the field. J Child Psychol Psychiatry. 2018. https://0-doi-org.brum.beds.ac.uk/10.1111/jcpp.12960.

  6. Schulte M, Ostroff C, Shmulyian S, Kinicki A. Organizational climate configurations: relationships to collective attitudes, customer satisfaction, and financial performance. J Appl Psychol. 2009;94:618–34.

    Article  PubMed  Google Scholar 

  7. Eccles MP, Mittman BS. Welcome to implementation science. Implement Sci. 2006;1:1.

    Article  PubMed Central  Google Scholar 

  8. Novins DK, Green AE, Legha RK, Aarons GA. Dissemination and implementation of evidence-based practices for child and adolescent mental health: a systematic review. J Am Acad Child Adolesc Psychiatry. 2013;52:1009–25.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Denison D. What is the difference between organizational culture and organizational climate? A native’s point of view on a decade of paradigm wars. Acad Manag Rev. 1996;21:519–654.

    Article  Google Scholar 

  10. Ostroff C, Kinicki AJ, Tamkins MM. Organizational culture and climate. In: Borman WC, Ilgen DR, Klimoski RJ, editors. Comprehensive handbook of psychology. New York: Wiley; 2003. p. 565–93.

    Google Scholar 

  11. Williams NJ, Ehrhart MG, Aarons GA, Marcus SC, Beidas RS. Linking molar organizational climate and strategic implementation climate to clinicians’ use of evidence-based psychotherapy techniques: cross-sectional and lagged analyses from a two-year observational study. Implement Sci. 2018;13:85.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Meyer AD, Tsui AS, Hinings C. Configurational approaches to organizational analysis. Acad Manag J. 1993;36:1175–95.

    Google Scholar 

  13. Ringeisen H, Henderson K, Hoagwood K. Context matters: schools and the “research to practice gap” in children's mental health. School Psych Rev. 2003;32:153.

    Google Scholar 

  14. Forman SG, Fagley NS, Steiner DD, Schneider K. Teaching evidence-based interventions: perceptions of influences on use in professional practice in school psychology. Train Educ Prof Psychol. 2009;3:226.

    Google Scholar 

  15. Farmer EM, Burns BJ, Phillips SD, Angold A, Costello EJ. Pathways into and through mental health services for children and adolescents. Psychiatr Serv. 2003;54:60–6.

    Article  PubMed  Google Scholar 

  16. Lyon AR, Ludwig KA, Vander Stoep A, Gudmundsen G, McCauley E. Patterns and predictors of mental healthcare utilization in schools and other service sectors among adolescents at risk for depression. School Ment Health. 2013;5:155–65.

    Article  Google Scholar 

  17. Fixsen D, Blase K, Metz A, Van Dyke M. Statewide implementation of evidence-based programs. Except Child. 2013;79:213–30.

    Article  Google Scholar 

  18. Odom SL, Cox AW, Brock ME, Asd NPDCO. Implementation science, professional development, and autism spectrum disorders. Excep Child. 2013;79:233–51.

    Article  Google Scholar 

  19. Pellecchia M, Connell JE, Beidas RS, Xie M, Marcus SC, Mandell DS. Dismantling the active ingredients of an intervention for children with autism. J Autism Dev Disord. 2015;45:2917–27.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Mandell DS, Stahmer AC, Shin S, Xie M, Reisinger E, Marcus SC. The role of treatment fidelity on outcomes during a randomized field trial of an autism intervention. Autism. 2013;17:281–95.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Suhrheinrich J, Stahmer AC, Reed S, Schreibman L, Reisinger E, Mandell D. Implementation challenges in translating pivotal response training into community settings. J Autism Dev Disord. 2013;43:2970–6.

    Article  PubMed  Google Scholar 

  22. Locke J, Olsen A, Wideman R, Downey MM, Kretzmann M, Kasari C, Mandell DS. A tangled web: the challenges of implementing an evidence-based social engagement intervention for children with autism in urban public school settings. Behav Ther. 2015;46:54–67.

    Article  PubMed  Google Scholar 

  23. Kang-Yi CD, Locke J, Marcus SC, Hadley TR, Mandell DS. School-based behavioral health service use and expenditures for children with autism and children with other disorders. Psychiatr Serv. 2015;67:101–6.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Glisson C, Landsverk J, Schoenwald S, Kelleher K, Hoagwood KE, Mayberg S. Research Network on Youth Mental Health. Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Admin Pol Ment Health 2008;35:98.

  25. Hartnell CA, Ou AY, Kinicki A. Organizational culture and organizational effectiveness: a meta-analytic investigation of the competing values framework's theoretical suppositions. J Appl Psychol. 2011;96:677.

    Article  PubMed  Google Scholar 

  26. Ostroff C, Kinicki AJ, Muhammad RS. Organizational culture and climate. In: Schmitt NW, Highhouse S, Weiner IB, editors. Handbook of psychology: industrial and organizational psychology. Hoboken: John Wiley & Sons Inc; 2013. p. 643–76.

    Google Scholar 

  27. Hofstede G, Neuijen B, Ohayv DD, Sanders G. Measuring organizational cultures: a qualitative and quantitative study across twenty states. Adm Sci Q. 1990;35:286–316.

    Article  Google Scholar 

  28. James LR, Choi CC, Ko CE, McNeil PK, Minton MK, Wright MA, Kim K. Organizational and psychological climate: a review of theory and research. Eur J Work Organ Psy. 2008;17:5–32.

    Article  Google Scholar 

  29. Beidas RS, Marcus S, Aarons GA, Hoagwood KE, Schoenwald S, Evans AC, Mandell DS. Predictors of community therapists’ use of therapy techniques in a large public mental health system. JAMA Pediatr. 2015;169:374–82.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Glisson C, Schoenwald SK, Kelleher K, Landsverk J, Hoagwood KE, Mayberg S. Research Network on Youth Mental Health. Therapist turnover and new program sustainability in mental health clinics as a function of organizational culture, climate, and service structure. Admin Pol Ment Health 2008;35:124–133.

  31. Glisson C, Williams NJ, Hemmelgarn A, Proctor EK, Green P. Increasing clinicians’ EBT exploration and preparation behavior in youth mental health services by changing organizational culture with ARC. Behav Res Ther. 2016;76:40–6.

    Article  PubMed  Google Scholar 

  32. Glisson C, Williams NJ. Assessing and changing organizational social contexts for effective mental health services. Annu Rev Public Health. 2015;36:507–23.

    Article  PubMed  Google Scholar 

  33. Olin SS, Williams NJ, Pollock M, Armusewicz K, Kutash K, Glisson C, Hoagwood KE. Quality indicators for family support services and their relationship to organizational social context. Admin Pol Ment Health. 2014;41:43–54.

    Article  Google Scholar 

  34. Williams NJ, Glisson C, Hemmelgarn A, Green P. Mechanisms of change in the ARC organizational strategy: increasing mental health clinicians’ EBP adoption through improved organizational culture and capacity. Admin Pol Ment Health. 2017;44:269–83.

    Article  Google Scholar 

  35. Glisson C, Green P. The effects of organizational culture and climate on the access to mental health care in child welfare and juvenile justice systems. Admin Pol Ment Health. 2006;33:433–48.

    Article  Google Scholar 

  36. Glisson C, Hemmelgarn A, Green P, Williams NJ. Randomized trial of the availability, responsiveness, and continuity (ARC) organizational intervention for improving youth outcomes in community mental health programs. J Am Acad Child Adolesc Psychiatry. 2013;52:493–500.

    Article  PubMed  Google Scholar 

  37. Williams NJ, Glisson C. Testing a theory of organizational culture, climate and youth outcomes in child welfare systems: a United States national study. Child Abuse Negl. 2014;38:757–67.

    Article  PubMed  Google Scholar 

  38. Woltmann EM, Whitley R, McHugo GJ, Brunette M, Torrey WC, Coots L, Drake RE. The role of staff turnover in the implementation of evidence-based practices in mental health care. Psychiatr Serv. 2008;59:732–7.

    Article  PubMed  Google Scholar 

  39. Harrison DA, Newman DA, Roth PL. How important are job attitudes? Meta-analytic comparisons of integrative behavioral outcomes and time sequences. Acad Manag J. 2006;49:305–25.

    Article  Google Scholar 

  40. Tett RP, Meyer JP. Job satisfaction, organizational commitment, turnover intention, and turnover: path analyses based on meta-analytic findings. Pers Psychol. 1993;46:259–93.

    Article  Google Scholar 

  41. Locke J, Beidas RS, Marcus S, Stahmer A, Aarons GA, Lyon AR, Mandell DS. A mixed methods study of individual and organizational factors that affect implementation of interventions for children with autism in public schools. Implement Sci. 2016;11:135.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Schreibman L. Intensive behavioral/psychoeducational treatments for autism: research needs and future directions. J Autism Dev Disord. 2000;30:373–8.

    Article  CAS  PubMed  Google Scholar 

  43. Smith T. Discrete trial training in the treatment of autism. Focus Autism Other Dev Disabl. 2001;16:86–92.

    Article  Google Scholar 

  44. Arick J, Loos L, Falco R, Krug DA. The STAR Program: strategies for teaching based on autism research. Austin: PRO-ED; 2004.

    Google Scholar 

  45. Stahmer AC. Using pivotal response training to facilitate appropriate play in children with autistic spectrum disorders. Child Lang Teach Ther. 1999;15:29–40.

    Article  Google Scholar 

  46. Dettmer S, Simpson RL, Myles BS, Ganz JB. The use of visual supports to facilitate transitions of students with autism. Focus Autism Other Dev Disabl. 2000;15:163–9.

    Article  Google Scholar 

  47. Pinnock H, Barwick M, Carpenter C, Eldridge S, Grandes G, Griffiths CJ, Rycroft-Malone J, Meissner P, Murray E, Patel A, Sheikh A, Taylor SJC. For the StaRI group. Standards for Reporting Implementation Studies (StaRI) statement. BMJ. 2017;356:i6795.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Glisson C, Green P, Williams NJ. Assessing the organizational social context (OSC) of child welfare systems: implications for research and practice. Child Abuse Negl. 2012;36:621–32.

    Article  PubMed  Google Scholar 

  49. Klein KJ, Dansereau F, Hall RJ. Levels issues in theory development, data collection, and analysis. Acad Manag J. 1994;19:195–229.

    Google Scholar 

  50. Chan D. Functional relations among constructs in the same content domain at different levels of analysis: a typology of composition models. J Appl Psychol. 1998;83:234–46.

    Article  Google Scholar 

  51. James LR, Demaree RG, Wolf G. RWG: an assessment of within-group interrater agreement. J Appl Psychol. 1993;78:306–9.

    Article  Google Scholar 

  52. Bliese PD. Within-group agreement, non-independence, and reliability: implications for data aggregation and analysis. In: Klein KJ, Kozlowski SWJ, editors. Multilevel theory, research, and methods in organizations: foundations, extensions, and new directions. San Francisco: Jossey-Bass; 2000. p. 349–81.

    Google Scholar 

  53. LeBreton JM, Senter JL. Answers to 20 questions about interrater reliability and interrater agreement. Organ Res Methods. 2008;11:815–52.

    Article  Google Scholar 

  54. Glisson C, Williams NJ, Green P, Hemmelgarn A, Hoagwood K. The organizational social context of mental health Medicaid waiver programs with family support services: implications for research and practice. Admin Pol Ment Health. 2014;41:32–42.

    Article  Google Scholar 

  55. Glisson C, Durick M. Predictors of job satisfaction and organizational commitment in human service organizations. Adm Sci Q. 1988;33:61–81.

    Article  Google Scholar 

  56. Stahmer AC, Reed S, Lee E, Reisinger EM, Mandell DS, Connell JE. Training teachers to use evidence-based practices for autism: examining procedural implementation fidelity. Psychol Sch. 2015;52:181–95.

    Article  PubMed  Google Scholar 

  57. Pellecchia M, Beidas RS, Marcus SC, Fishman J, Kimberly JR, Cannuscio CC, Mandell DS. Study protocol: implementation of a computer-assisted intervention for autism in schools: a hybrid type II cluster randomized effectiveness-implementation trial. Implement Sci. 2016;11:154.

    Article  PubMed  PubMed Central  Google Scholar 

  58. Bakk Z, Tekle FT, Vermunt JK. Estimating the association between latent class membership and external variables using bias adjusted three-step approaches. Sociol Methodol. 2013;43:272–311.

    Article  Google Scholar 

  59. Bolck A, Croon M, Hagenaars J. Estimating latent structure models with categorical variables: one-step versus three-step estimators. Polit Anal. 2014;12:3–27.

    Article  Google Scholar 

  60. Bakk Z, Vermunt JK. Robustness of stepwise latent class modeling with continuous distal outcomes. Struct Equ Model. 2016;23:20–31.

    Article  Google Scholar 

  61. Vermunt JK, Magidson J. Latent class cluster analysis. In: Hagenaars JA, McCutcheon AL, editors. Applied latent class analysis. Cambridge: Cambridge University Press; 2002. p. 89–106.

    Chapter  Google Scholar 

  62. Nylund KL, Asparouhov T, Muthen BO. Deciding on the number of classes in latent class analysis and growth mixture modeling: a Monte Carlo simulation study. Struct Equ Modeling. 2007;14:535–69.

    Article  Google Scholar 

  63. Raftery AE. Bayesian model selection in social research. Sociol Methodol. 1995;25:111–63.

    Article  Google Scholar 

  64. Celeux G, Soromenho G. An entropy criterion for assessing the number of clusters in a mixture model. J Classif. 1996;13:195–212.

    Article  Google Scholar 

  65. Foti RJ, Thompson NJ, Allgood SF. The pattern-oriented approach: a framework for the experience of work. Ind Organ Psychol. 2011;4:122–5.

    Article  Google Scholar 

  66. Morin AJ, Morizot J, Boudrias JS, Madore I. A multifoci person-centered perspective on workplace affective commitment: a latent profile/factor mixture analysis. Organ Res Methods. 2011;14:58–90.

    Article  Google Scholar 

  67. Vermunt JK. Latent class modeling with covariates: two improved three-step approaches. Polit Anal. 2010;18:450–69.

    Article  Google Scholar 

  68. Muthén LK, Muthén BO. Mplus user’s guide: statistical analysis with latent variables. Muthén & Muthén: Los Angeles; 1998.

    Google Scholar 

  69. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale: Lawrence Earlbaum Associates; 1988.

    Google Scholar 

  70. Henggeler SW, Chapman JE, Rowland MD, Halliday-Boykins CA, Randall J, Shackelford J, Schoenwald SK. Statewide adoption and initial implementation of contingency management for substance-abusing adolescents. J Consult Psychol. 2008;76:556–67.

    Article  Google Scholar 

  71. Ostroff C, Schulte M. A configural approach to the study of organizational culture and climate. In: Schneider B, Barbera KM, editors. The Oxford Handbook of Organizational Climate and Culture. New York: Oxford University Press; 2014. p. 532–52.

    Google Scholar 

  72. Schneider CQ, Wagemann C. Set-theoretic methods for the social sciences: a guide to qualitative comparative analysis. New York: Cambridge University Press; 2012.

    Book  Google Scholar 

  73. McKay PF, Avery DR, Liao H, Morris MA. Does diversity climate lead to customer satisfaction? It depends on the service climate and business unit demography. Organ Sci. 2011;22:788–803.

    Article  Google Scholar 

  74. Ehrhart MG, Aarons GA, Farahnak LR. Assessing the organizational context for EBP implementation: the development and validity testing of the implementation climate scale (ICS). Implement Sci. 2014;9:157.

    Article  PubMed  PubMed Central  Google Scholar 

  75. Ryan RM, Deci EL. Self-determination theory: basic psychological needs in motivation, development, and wellness. Guilford: New York; 2017.

    Google Scholar 

  76. Williams NJ, Beidas RS. Navigating the storm: how proficient organizational culture promotes clinician retention in the shift to evidence-based practice. PLoS One. 2018;13:e0209745.

    Article  PubMed  PubMed Central  Google Scholar 

  77. Glisson C. The organizational context of children’s mental health services. Clin Child Fam Psychol Rev. 2002;5:233–53.

    Article  PubMed  Google Scholar 

  78. Rollins AL, Salyers MP, Tsai J, Lydick JM. Staff turnover in statewide implementation of ACT: relationship with ACT fidelity and other team characteristics. Admin Pol Ment Health. 2010;37:417–26.

    Article  Google Scholar 

  79. Hom PW, Lee TW, Shaw JD, Hausknecht JP. One hundred years of employee turnover theory and research. J Appl Psychol. 2017;102:530–45.

    Article  PubMed  Google Scholar 

  80. Aarons GA, Ehrhart MG, Farahnak LR, Sklar M. Aligning leadership across systems and organizations to develop strategic climate for evidence-based practice implementation. Annu Rev Public Health. 2014;35:255–74.

    Article  PubMed  PubMed Central  Google Scholar 

  81. Aarons GA, Ehrhart MG, Farahnak LR, Hurlburt MS. Leadership and organizational change for implementation (LOCI): a randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implement Sci. 2015;10(1):11.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We are grateful for the collaboration and assistance of our school partners whose support made this work possible. We also wish to thank Dr. Steven Marcus for his consultation on this project.

Funding

This study was supported by the following grants from the US National Institute of Mental Health: K01 MH100199 (PI: Locke), P50 MH113840 (PIs: Beidas, Mandell, Volpp), R01MH106175 (PI: Mandell), and F31MH112211 (PI: Frank).

Availability of data and materials

The data used in this manuscript are available upon reasonable request. Please contact the senior author (JL) for more information.

Author information

Authors and Affiliations

  1. School of Social Work, Boise State University, 1910 University Drive, Boise, ID, 83625, USA

    Nathaniel J. Williams

  2. Department of Psychology, Temple University, 1701 N 13th St., Philadelphia, PA, 19122, USA

    Hannah E. Frank

  3. Department of Speech and Hearing Sciences, University of Washington, 1417 NE 42nd St, Seattle, WA, 98105, USA

    Lindsay Frederick & Jill Locke

  4. Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, 3535 Market Street, 3rd floor, Philadelphia, PA, 19104, USA

    Rinad S. Beidas & David S. Mandell

  5. Department of Medical Ethics and Health Policy, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA

    Rinad S. Beidas

  6. Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, PA, USA

    Rinad S. Beidas

  7. Department of Psychiatry, University of California, San Diego, CA, USA

    Gregory A. Aarons

  8. Center for Behavioral Health Research, University of Tennessee, Knoxville, TN, USA

    Philip Green

Authors
  1. Nathaniel J. Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hannah E. Frank
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lindsay Frederick
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rinad S. Beidas
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David S. Mandell
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Gregory A. Aarons
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Philip Green
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jill Locke
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

NJW and JL generated the idea and designed the study. NJW led all quantitative data analysis and contributed the majority of the writing. HF and LF drafted sections of the manuscript and provided input in the interpretation of the results. Authors RB and DM provided input into the design and conceptualization of the study. PG analyzed data from the OSC measure. JL drafted sections of the manuscript and is the principal investigator of the study from which these data are drawn. All authors were involved in developing, editing, reviewing, and providing feedback for this manuscript and have given approval of the final version to be published.

Corresponding author

Correspondence to Nathaniel J. Williams.

Ethics declarations

Ethics approval and consent to participate

The University of Pennsylvania and the University of Washington Institutional Review Boards approved this study (Protocol #820870).

Consent for publication

Not applicable.

Competing interests

GAA is an associate editor of Implementation Science; all decisions on this paper were made by a different editor. All authors declare that they have no other competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

Standards for Reporting Implementation Studies: the StaRI checklist for completion. (DOCX 78 kb)

Additional file 2:

Table S1. Raw Correlation Matrix of OSC Dimensions and Fidelity Measures. Table S2. OLS Regression Analyses Linking OSC Dimensions to Evidence-Based Practice Fidelity. (DOCX 43 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Williams, N.J., Frank, H.E., Frederick, L. et al. Organizational culture and climate profiles: relationships with fidelity to three evidence-based practices for autism in elementary schools. Implementation Sci 14, 15 (2019). https://0-doi-org.brum.beds.ac.uk/10.1186/s13012-019-0863-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s13012-019-0863-9

Keywords

Implementation Science

ISSN: 1748-5908