What You Need to Know About the New Surgeon Scorecard App

In need of a knee replacement, prostate resection or gall bladder removal? Now, you can search the Surgeon Scorecard, a new app from the nonprofit newsroom ProPublica, that lists the complication rates of surgeons across the United States.

ProPublica found half of all hospitals in America have uneven performances among their surgical staff: They have some surgeons with low complication rates and some with high rates. Therefore, it may not be enough for patients to select a well-respected academic hospital and simply hope for the best.

At first glance, the app is an impressive feat of data journalism, a three-year project. It includes just shy of 17,000 surgeons, and pulls data from more than 63,000 Medicare patients who were readmitted with complications between 2009 and 2013. The database also includes roughly 3,400 deaths during that period.

The ProPublica team of Marshall Allen, Olga Pierce and Sisi Wei told KQED they crunched an expansive data set from the federal Centers for Medicare and Medicaid Services, which administers the programs. Then they cross-referenced that data with a number of other sources. CMS agreed to provide this data to ProPublica, with strict limits on sharing the raw data to protect patient privacy.

The app includes information about some limitations of the data and about the methodology used to calculate surgical complication rates. The authors stress that they took a conservative approach -- no rate is reported if the surgeon performed that procedure fewer than 20 times, for instance.

"We wouldn't have been able to come up with our methods alone," Pierce said. "We were helped a lot by people who advised us on the best way to use the data." The help included a panel of medical experts.

How Should You Use the App? 

Pierce and Wei say the app is already proving popular with patients, but they declined to disclose how many are using it. They expect that, like other health-transparency tools, it will gain traffic over time.

The first thing the ProPublica team wants you to know is the app is not designed as a be-all and end-all solution. The team recommends patients and their family members use the app as a starting point when researching to find the right surgeon. Some patients may have a gut instinct about a physician, and all patients benefit from asking deeper questions before committing to a procedure.

The team said they hope the Surgeon Scorecard will prompt patients and surgeons to have a frank conversation about potential complications. Wei recommends patients ask their surgeon how many procedures they've performed and whether they know their own complication rate.

But don't try to catch your surgeon in a lie by comparing your surgeon's response to the scorecard. The Surgeon Scorecard is by no means definitive and includes some gaping holes. CMS barred ProPublica from reporting the exact number of complications if it's between one and ten. The app also doesn't include hospital stays paid for by private insurance or other government programs, like Medicaid.

Patient advocates stressed to KQED that the Surgeon Scorecard is just one data point to combine with many others, including transparency tools, online community forums and more. For example, Regina Holliday, a medical advocate from Grantsville, Maryland, suggests patients find out what kind of rehabilitation services are provided after a knee or hip replacement.

Patients should also do some digging about whether it's worth getting the procedure at all, particularly if they've heard from only one doctor. And, as Holliday points out, the data don't touch on whether the procedure was successful from the point of view of former patients.

Is it Fair to Surgeons?

The scorecard has gotten positive and negative reviews from the medical community. Some physicians vehemently believe it's not useful to report a surgeon's history of complication rates at all -- and that the scorecard should be retracted.

Meanwhile, one anonymous physician raved about the Surgeon Scorecard to a health care blog, arguing it would set them apart from less talented rivals who were reaping in cash.

Others take a more nuanced view.

Dr. Bob Wachter, Professor and Interim Chairman of the Department of Medicine at the University of California, San Francisco, said measuring quality is an imperfect science. He does acknowledge, though, that it is useful for patients.

In an interview, Dr. Wachter raised a number of concerns about how surgeons are portrayed in the app:

• It's not always the surgeon's fault: The outcome of a procedure may have less to do with the surgeon and more to do with the hospital system in which they work (which includes the pharmacists, nurses and hospital leadership.) Moreover, a bad outcome may be a result of a patient failing to take their medication due to a lack of support or a psychiatric condition.
• Some surgeons take on risky cases: The data may be skewed against surgeons who take on tricky cases, which are more likely to lead to bad outcomes. It requires a lot of data about a particular case to make sense of the ultimate outcome.
• Some surgeons work at hospitals that care for the very sick and the very poor: Some hospitals are regularly referred patients that no others will take. Some patients may also lack social services after they've been discharged, and are more likely to be readmitted.
• Some surgeons have a conservative practice style: They may readmit patients more regularly just to monitor them. Although, it may be possible to screen for this by checking the length of the hospital stay.

The team at ProPublica did acknowledge these points and said they took pains to mitigate them in the following ways:

• They did not list complications when they spotted patients who "looked unusual in some way." They also scanned patient records for evidence of comorbidities, like obesity and diabetes -- and did not include these patients in the database.
• They factored the "hospital effect" into the model, meaning that if a hospital is known to release patients without thorough discharge instructions, it may not be the surgeon's fault.
• They examined the number of days when a patient was readmitted. If the bulk of those stays were three days or longer, that suggested a deeper complication, rather than a conservative approach.
• They factored into their model the socio-economic status of patients, using SSI data. There are still be some gaps in this data, however, like the patient's mental state.

The reporters approached the project with the philosophy that surgeons' actions have a major effect on patient outcomes -- and therefore the good and bad actors should be called out. They reference the American College of Surgeons' statement of principles, saying surgeons are responsible for a patient’s entire course of care.

"I will say that surgeons may not be directly responsible, but they do have a lot of power to address these things," Pierce said.

Other Limitations and Concerns

The most notable hole in ProPublica's data is that they only include Medicare patients. Another important limitation is their reliance on procedure codes. There is a great deal of variation in how hospitals code, including some that even coded the wrong surgery. 

As Wachter points out, another limitation is that some complications do not lead to readmission or death, and therefore may not be included in the Surgeon Scorecard.

Another widely-held concern among doctors and patient advocates is that some physicians may view the Surgeon Scorecard as a benchmark -- if it continues to be popular -- and will opt against taking on tricky cases.

"It's possible that surgeons get frustrated and don't want to stick their neck out in the future," said Julia Hallisy, a San Francisco-based patient advocate.

Next Steps

Limitations aside, it's important to note that a project like this one would not have been possible a decade ago when hospitals across the country were still relying on paper records. With health systems slowly digitizing, such transparency projects that include thousands, if not millions, of data-sets are now feasible.

ProPublica is currently negotiating with states to release similar data, which would include all payers. Currently, it's difficult to follow a specific patient across hospital visits when they have different insurers.

In coming months, the team plans to update the data-set with more recent complication rates, and make changes based on the initial feedback and concerns. They're eager to hear from you.

Have you used the Surgeon Scorecard? Got any concerns to share? Contact the team on Twitter using the hashtag #surgeonscorecard.