Neural Nets Improve Hospital Treatment Quality and Reduce Expenses June 9, 1990 by Jeannette Lawrence A new hospital information and patient prediction system has improved the quality of care, reduced the death rates and saved millions of dollars in resources at Anderson Memorial Hospital in South Carolina. The CRTS/QURI system uses neural networks trained with BrainMaker (from California Scientific Software) to predict the severity of illness and use of hospital resources. Developed by Steven Epstein, Director of Systems Development and Data Research, the CRTS/QURI system's goal is to provide educational information and feedback to physicians and others to improve resource efficiency and patient care quality. The first study showed that the program was directly responsible for saving half a million dollars in the first 15 months even though the program only included half of the physicians and three diagnoses. Since then, the number of diagnoses and physicians included in the program have increased. The quality of care has improved such that there are fewer deaths, fewer complications, and a lowered readmission rate. Expenses have been reduced by fewer unnecessary tests and procedures, lowered length of stays, and procedural changes. Over the past five months the hospital's accounting department has reported a savings in the millions of dollars, although it is difficult to say exactly how much of this is directly due to CRTS/QURI. According to Epstein, "The hospital may now be experiencing a "spillover effect" of new physician knowledge and behavior into areas not initially targeted by the CRTS/QURI system." The reported success has motivated several other hospitals to join in the program and has provided the impetus to begin a quality program with the state of South Carolina. At the core of this new system lies several back-propagation neural networks which were designed using the BrainMaker program on a PC. Individually trained neural networks learn how to classify and predict the severity of illness for particular diagnoses so that quality and cost issues can be addressed fairly. After attempts to use regression analysis to predict severity levels for several diagnoses failed, Epstein turned to the BrainMaker program for a new approach and taught his neural networks to classify and predict severity with 95% accuracy. The neural networks are also used to predict the mode of discharge - routine through death - for particular diagnoses. Without these neural networks, the system could not automatically compare cases with the same diagnoses and like severity levels. Most commercially available severity programs provide mortality as the only measure of quality. Physicians typically reject systems which only compare costs or mortality rates as being too crude or unfair. In order to provide more meaningful feedback and valid comparisons within homogeneous groups of patients, the CRTS Acuity Indexing Scale was developed. Patients are rated for the severity of illness on a scale from -3 to +3. Training information is based upon the length of stay in the hospital which has a direct relationship to the severity of the illness (acuity). When making predictions, a -3 patient is expected to require the least hospitalization and resources, a 0 patient average, and a +3 patient the most. Cases which are run through the trained network can then be indexed in a database by their severity level and retrieved by such for comparisons. Neural Network Designs The neural network was trained to make the severity prediction using variables of seven major types: diagnosis, complications/comorbity, body systems involved (e.g., cardiac and respiratory), procedure codes and their relationships (surgical or nonsurgical), general health indicators (smoking, obesity, anemia, etc.), patient demographics (race, age, sex, etc.), and admission category. Using a neural network to learn the effects that these variables have on the severity of illness takes the CRTS/QURI a step beyond the other programs available for indexing severity of illness. With a neural network, it is possible to discover relationships that were previously unrecognized. There is no need to define how the variables are related to the output, the neural network learns it on its own. BrainMaker provides the ability to view how any one of these variables effects the output of the network. Using BrainMaker's neuron graphics any input can be varied over some range and the effect that this has can be plotted. The neural network learns (just as profession literature states) that elderly patients with the same diagnosis require more care than younger patients and that patients with family support tend to require shorter hospital stays. Unlike traditional programs, there is no need to translate premises like these into program statements or formulas. The neural network learns these and other associations by viewing case after case until it picks up the patterns using the back-propagation algorithm. Three years of patient data was chosen for training. There were approximately 80,000 patients to choose from and 473 primary diagnoses. The cases were pre-selected with multiple regression techniques to eliminate outliers so that no "bizarre" cases were used for training the neural network. For a given diagnosis, about 400 to 1000 cases were used to train on. Data was collected from automated medical records, the QURI program (explained in the next section), and Federally required files for Medicare patients. These were downloaded and read into the CRTS database which is in an rBase format on a PC. The selected training data was output from the rBase file as a dBase file because there is a direct link to BrainMaker this way. The dBase file was read by BrainMaker's preprocessor called NetMaker which automatically translates the data into neural network files for BrainMaker. These files define the network architecture, the screen display, and the training and testing data in BrainMaker format. After these automatic conversions were done, the neural network was trained in BrainMaker. During the initial neural network design phase, decisions had to be made as to which variables were important so that unnecessary data collection could be avoided. In order to help make this decision, Epstein used BrainMaker's Hinton diagrams to see which inputs affected the trained network the most. These diagrams present a picture of connection strengths (weights) at the hidden and output layers. A neuron which connects to all the neurons in the next layer with the same strength transmits no useful information. Two neural networks for each diagnosis were trained - one to predict the use of resources and the other to predict the type of discharge. For a single diagnosis network, there are 26 input variables and one output variable. BrainMaker trained in about 4 hours using a NorthGate 386 running at 33 MHz. Epstein began training with a .1 tolerance which means every prediction for every case must be 90% accurate. Then he lowered the tolerance to .085 and eventually stopped with the network trained at .05 training tolerance (95% accurate). Then a test set of cases was run through. The BrainMaker program has built-in testing capability. In addition, Epstein and Jones verified the results of the network before using the network to categorize patients and make predictions. The CRTS/QURI System Once the cases are indexed according to severity level, the CRTS (Computerized Resource Tracking System) can be used to provide educational feedback to physicians concerning quality care issues and the use of hospital resources. CRTS combines computer software which produces automated graphs, reports and summaries along with a comprehensive educational format. An artificial intelligence program, VP Expert, is used to generate individual physician's reports. The CRTS program is more affordable than other indexing systems in that no additional chart abstracting is necessary. The majority of information is retrieved from the hospital's "mainframe" computer. Information sources include the medical records abstract, UB82 files (medicare), and QURI data. The "QURI" portion of the CRTS/QURI system is an integrated data collection and reporting program. Epstein developed this stand-alone menu-driven, user-friendly software program which provides computer integration for departments which provide additional data for quality analysis. QURI stands for Quality assurance, Utilization review, Risk management and Infection control. Quality Assurance monitors various criteria for specific procedures and diseases. Utilization Review monitors resource related items such as admission justification and intensity of services. Risk Management determines things that put a patient or hospital at risk (such as a fall out of bed), and Infection Control monitors infections acquired during the hospital stay. CRTS is used for evaluation at various levels. At the hospital level, comparisons are made to data published by the profession to identify areas of major concern for the hospital. Individual diagnoses that show problems of quality and/or finance are identified. At the physician specialty group level, treatment for various severity levels of a diagnosis are discussed. At the individual physician level, one page of written summary and five pages of comparative graphs are provided. This confidential report shows how a physician's patients compared to the hospital wide average for a particular diagnosis. Comparisons are provided for areas such as the types of procedures used, ancillary services used, patient demographics, infection rate, complications, readmissions, and predicted (versus actual) illness severity level, charges, and mortality rate. A summary of problem areas and a list of suggestions are also included in the report. CRTS is implemented in two primary modules 1) collecting and analyzing the data with neural networks, and 2) conducting the physicians' education program. Epstein, who originally designed the system for his dissertation, is in charge of the data-related portions. Dr. Fred Jones, Executive Vice President of Medical Affairs at Anderson Memorial, sets up physician education (feedback) programs, trains them in the use of system, and verifies trained neural networks results. Physician Response Getting the physicians to accept the new system was no easy task. "Initially some of the physicians must have thought the program was a Communist Plot. One even stood up in an early meeting to declare the whole thing 'crap'," says Epstein. He explains that Jones has done a tremendous job in getting the physicians involved. The physicians are given a say as to what information is important, what should be in the reports, and how to use the data. Epstein continues, "The problem we're having now is that we're OVERWHELMED with requests for more diagnosis studies and more special reports from the physicians." What Can Be Learned from CRTS/QURI The QURI system includes the following components: infection surveillance, drug use evaluation, blood usage review, surgical case review, monitoring and evaluation, physician demographics, case review/peer review, incident/occurrence monitoring, claims management and utilization review. This information, as well as information from patient's charts is used to provide reports, graphs and charts to individual physicians or specialty groups. Hospital-wide comparisons can be trended over time and be compared to statewide averages or averages reported in the literature when available. Comparisons between the 473 diagnoses can be evaluated and compared to literature or statewide averages when available. Using the severity groups, computer generated graphs and charts can be presented to physician specialty groups and/or individual physicians displaying how they compare to their average hospitalwide peer or to their average within a specialty group for many resource and quality issues (see Comparative Information sidebar). Cases with similar severity levels can be found and treatments compared. Trends are also revealed in these same graphs depicting how a physician compares throughout severity levels. A physician or group may practice more efficiently and with improved quality when treating higher severity level patients. More indepth studies can be provided, such as how a physician's expired patient with a certain severity level was treated, what procedures and services were used, etc., versus other similarly ill patients. During a report review a question may come up such as why did Risk Management note that there was a problem? One possible scenario might be that the patient fell out of bed and a closer look at the report would show if Quality assurance reported that too much sedation was given. One report from the CRTS brings it all together. How It Began at Anderson Memorial The first diagnosis chosen for the program was DRG89 - pleurisy/ pneumonia. Compared to the national norm, the hospital mortality rate was higher. In addition, the hospital was losing money from cost overruns since insurance companies and medicare coverage have limits on what they will pay. It was learned that pneumonia of unknown etiology had the highest mortality. The reports showed that patients of family practitioners had longer lengths of stay. A typical scenario unfolded. Physicians would start their patients on antibiotics, wait for lab results (which were inconclusive more than 25% of the time) then ask for an internal medicine consult on the third day. Then treatment started all over. These patients stayed an average of 4 days longer and the hospital averaged a $2764 loss. Since this discovery, Family Practitioners now get an earlier consult. Further research found that the sputum collection procedure was a problem. The hospital now uses respiratory therapy to collect specimens. Samples are quickly tested and thrown out if they are spit instead of sputum from the lungs. The inadequate sputum rate decreased from 26% to 5%. The improvements caused a drop of $1091 in treating an average case of pneumonia during the pilot program even though only half of the physicians were involved in the program. Other improvements for DRG89 followed. The length of stay and charges continued to fall through 1989. Complications and mortality rates also decreased. The use of ancillary services and ICU decreased. When another diagnosis was added to the program, cerebovascular disease, the hospital experienced a decrease in total charge of $1043 per patient, even though CAT scans increased. Average length of stay also decreased. Anderson Memorial is now using nine diagnoses in its program. What's Downstream In response to the national concern for the quality of health care, the South Carolina Hospital Association voted to approve a statewide Quality Indicators Project with the state. The tracking method was developed by the Systems Development and Data Research department at Anderson Memorial. Epstein is currently working with the state of South Carolina to develop a software package which will look at quality issues for statewide comparisons. Seen as only the beginning, Epstein hopes to use the results of this statewide quality project as a basis for designing another neural network for use in the hospital. This neural network would screen all patient charts and other data to provide a quality index (based upon the statewide issues such as mortality, infections, etc.). He proposes that the results could be used to decide which cases should be reviewed, instead of randomly picking charts to look at. "There are too many patients and too much data for someone to look at every single one and decide which ones have quality issues that need looking into with a peer review," Epstein says. This new neural network would make educating physicians a much more efficient process as well as improve the quality of care even further. Currently, Anderson Memorial is working with three other hospitals as they start their own program. A recent presentation of the CRTS/QURI system at an American College of Physician Executives conference produced an overwhelming response. The hospital has been beseiged with requests for more information from other interested hospitals. In the very near future at a meeting with seven hospitals from a hospital network, Epstein will propose a hospital-group program. In this program, there would be one neural network trained for each hospital, plus one for all the hospitals using one diagnosis initially. An overall advisory group would be formed. Comparisons could be made between hospitals, between physicians and the seven-hospital average, as well as all the currently implemented comparisons. If approved, he plans to start the program in a few months. After a six-month pilot program, results will be checked and verified. And this is only the beginning. ***************************************************************************** The CRTS program can provide comparative information for the following: Resource Indicators Quality Indicators ------------------- ------------------ total patient charge inpatient mortality length of stay unplanned readmissions ancillary charge hospital-acquired infections ancillary ratio surgical-wound infections ICU days neonatal mortality number of consultants perioperative mortality cardiopulmonary charges Cesaerean sections physical therapy charges unplanned returns to special laboratory charges care unit radiology charges unplanned returns to O.R. catscan charges pharmacy charges IV solutions required ***************************************************************************** Note: The bulk of this material is expected to be printed in PCAI approximately November, 1990.