The study draws on commercially insured patients who had encounters with 1,444 adult PCPs from 27 medical groups in California between the years 2003–2006. During each of the four survey years (2004–2007), a random sample of approximately 100 patients per physician who had at least one visit with their PCP during the prior year were mailed a survey. All commercially insured patients visiting the physicians during the study year were eligible to receive a survey. Although an individual patient could respond to the survey in multiple years, the deidentified patient survey data could not be linked at the patient level.

The survey included core measures from the Clinician & Group CAHPS survey23 and supplemental measures from the Ambulatory Care Experiences Survey (ACES), a validated survey that measures patients’ experiences with a specific, named physician and that physician’s practice24. Mailings included an invitation letter, a printed survey, and a postage-paid return envelope. The survey invitation included a personal online code that gave respondents the option of completing the survey using the web. Previous analysis has demonstrated the absence of web survey mode effects for the patient survey questions25. The survey invitation listed a toll-free number for patients to obtain surveys in Spanish. A second invitation and questionnaire were sent to non-respondents 2 weeks after the initial mailing. Each annual data collection effort spanned a period of approximately 8 weeks.

For this analysis, we consider four survey composite measures: physician communication (six items), care coordination (two items), access to care (five items), and office staff interaction (two items) (see Appendix). The physician communication, access to care, and office staff interaction summary measures represent core item and composite content of the CAHPS Clinician & Group Survey23, which was endorsed by the National Quality Forum (NQF) for use in evaluating ambulatory care received from individual physicians and their practices. All survey questions consist of a six-point response continuum ranging from “never” to “always”. All questions reference care received from that particular physician and the physician’s practice over the past 12 months. Composite measures all achieve physician-level reliability of 0.70 or higher with samples of approximately 30–40 established patients per physician24,26,27.

As detailed elsewhere24, survey questions were scored linearly from 0 (“never”) to 100 (“always”) points, with higher scores indicating more favorable performance. Composites were calculated as the unweighted average of responses to all items comprising the measure after applying the half-scale rule28,29, which includes only respondents who complete at last half of the questions comprising a composite. Most measures had consistent item consistent and wording across years. For the access to care composite, we used an adjusted half-scale approach to calculate scores30. This scoring method facilitated the comparison of composites across time because items were treated in a comparable way across years, and each individual item’s contribution to the overall scale was factored into the composite calculation.

Of 399,392 outgoing patient surveys over the 4 study years, 14,226(3.6%) were undeliverable because of bad address information or patient death. Surveys were received from 145,522 respondents, yielding an adjusted response rate of 37.8%. The analytic sample included 135,401 respondents (average per physician = 93.8) who confirmed having seen their PCP during the prior 12 months. Respondents who did not confirm the named physician as their PCP or indicated that they did not visit the physician during the prior 12 months (n = 10,121) were excluded from the analysis. The commericially insured respondent sample was 35.1% male, 64.9% non-Hispanic White, 13.1% Hispanic, 12.2% Asian, 3.6% Black, 44.4% reported completing college, and 67.5% reported having at least one chronic medical condition; 47.1% reported being established with their PCP for 5 years or longer. Of the 1,444 physicians included in the analytic sample, 427 physicians participated in the survey initiative all 4 years (2004–2007), 283 physicians participated during 3 of the 4 years, and 734 participated during 2 of the 4 years.

Medical director interviews were conducted via telephone between April and June 2007. The pool of eligible medical groups consisted of groups that participated in IHA’s medical group performance-based financial incentive program and assessed patient care experiences at the individual physician-level in 2007. The interview assessed whether PCPs were eligible to receive performance-based financial incentives and the maximum possible magnitude of the incentives as a percentage of base compensation. In addition, directors were asked about the formulae used to calculate physician incentives, including the percent of the incentive that was based on productivity (e.g., average patients seen per day), efficiency (e.g., limiting referrals, effective panel management), patient experience measures, clinical quality measures, and “other” criteria (physician seniority, prescribing generic medications, and other contributions to the medical group). Finally, directors were asked whether their organization was presently engaged in various patient experience performance improvement activities, including sharing patients’ experience measures with physicians in individual feedback sessions, interpersonal skills training, business practice redesign, and practice leader compensation. A composite measure reflecting the total number of patient experience improvement activities adopted by groups was constructed by summing responses to the individual questions (range: 0–4, α = 0.65). Of the 27 medical groups eligible for this study (groups with 2 or more years of participation in the patient survey initiative), interviews were conducted with 25 medical group directors or designees, resulting in a 92.6% response rate.

Change over time for each patient survey composite was assessed using two sets of multilevel regression models. First, models were specified for each patient survey composite that used physician and medical group random effects to account for the clustering of patients within physicians and physicians and medical groups. These unadjusted models included a continuous measure of time (survey year). The second set of multilevel regression models was specified identically to the unadjusted models, but also controlled for patient age, gender, race/ethnicity, education, and self-rated physical health, which are commonly used to adjust patient experience measures31,32. For the physician random effects, the intercept value represents the first year of participation in the survey initiative for each physician, and the slope represents the annual performance change over the physician’s baseline year. The model also permitted calculation of the correlation between the physician intercept and slope33 in order to assess the extent to which physician performance change over time was associated with baseline performance. Following Elliott et. al.34, we standardized the change over time coefficients in terms of physician-level standard deviations (SDs) for each composite. This allowed us to describe the average annual change as the change in percentile points it would represent for a physician at the 50th percentile at baseline. We also examined the extent to which alternative explanations, including panel maturation, varying (and sometimes small) patient samples sizes per physician, and physicians’ years of participation might account for change over time. These analyses are presented in the online appendix.

Physicians’ baseline CAHPS composite scores were compared by medical group characteristics, financial incentives, and patient experience improvement activities. Differences in medical group baseline composite scores by medical group variables were assessed using multilevel models that accounted for the clustering of patients within physicians and physicians within medical groups. Finally, for each survey composite measure, we assessed the medical group-level predictors of physician performance change over time. We chose to introduce each of the medical group financial incentive and patient experience improvement variables independently because the modest number of medical groups involved (n = 25) could result in overfitting of models. For example, 24 degrees of freedom at the medical group level could perfectly capture any arbitrary pattern of the medical group means, even if they were entirely unrelated to the specified predictors. Each model was specified using physician and medical group random effects and controlled for medical group type (staff model vs. IPA vs. hybrid). To account for differences in the patient case mix across individual physician practices and years, models also controlled for patient age, gender, race/ethnicity, education, and self-rated physical health. Each model included the main effect and time interactions for the medical group variable under study. Statistically significant medical group variables were then specified jointly in models. Joint Wald tests were used to examine the extent to which the set of interaction terms (change over time predictors) jointly accounted for a statistically significant amount of the observed change over time on the composite measure. All continuous patient and medical group measures were standardized to a mean of 0 and a variance of 1 for the comparable interpretation of regression coefficients. All analyses were conducted using the XTMIXED module in STATA 10.033, which uses restricted maximum likelihood (REML) to estimate random effects.

When significant medical group predictors of change over time were jointly tested in models that included one another, most joint tests indicated that the combinations remained statistically significant. This suggests that one or more of the variables included in the joint test explain change over time on the patient experience composite, but multilevel models that account for both interaction terms lack the statistical power to distinguish between them. For example, the magnitude and formulae of individual physician financial incentives were independently associated with improvements on the physician communication composite, but not in models that included one another. Tests examining the joint significance of financial incentive magnitude and formulae (χ²= 11.59, p < 0.05) suggest a substantial amount of shared association with improvements in physician communication.