Rationale and Methodology

A well-functioning higher education market requires transparent information on costs and outcomes. Consumers looking to invest time and money in a college education should have access to information about costs and quality that is consistent and comparable across institutions and programs of study. Citizens and public officials also need information on the performance of their state’s higher education system in order to hold decision makers accountable.

At a minimum, transparency requires that institutions collect and publish basic data on degrees produced, graduation rates, and tuition costs. But a commitment to transparency should extend further, to include new performance measures such as student learning and student success in the labor market. Measuring these important outcomes can also help to ensure that we are rewarding campuses that produce credentials with labor market value, not just those that crank out more diplomas. Additionally, in this cost-conscious era, higher education systems need to be increasingly transparent about costs and efficiency. For all these performance measures, states should endeavor to go beyond simple reporting to document how their postsecondary results compare with past performance, state goals, and top-performing states and the nation as a whole.

In this category, we rewarded states that are making an effort to measure student outcomes beyond degree completion and those that proactively provide information to the public about the performance of institutions and the system as a whole. To evaluate each indicator, we contacted officials in every state to ask about their transparency policies. We then followed up by searching through each state’s relevant higher education websites (whether run by a state higher education office, university system, or community college system). It is important to note that the policy environment in the states is fluid and constantly changing, making these metrics something of a moving target. In general, we focused on whether states had formally enacted and implemented these policies by mid-April 2012.

Transparency: Public Accountability.

We believe that higher education transparency has two distinct objectives. The first is to foster public accountability by providing stakeholders with information on the productivity, performance, and costs of the state’s postsecondary system.

To translate this concept into a measurable metric, we collected data on whether a state had mechanisms to report information about higher education performance and outcomes. We first asked state higher education agencies to direct us to whatever publications, websites, or other resources the state uses to inform stakeholders about the higher education system. We also scoured each state’s higher education website to locate these resources. Because many states have distinct governance structures for their four- and two-year colleges, we coded public accountability for four- and two-year institutions separately.

We then scored states according to whether they report information on basic student outcomes such as graduation rates; whether the state reports additional outcome measures such as student learning or postgraduation success; and whether the state reports measures of institutional efficiency and cost- effectiveness. When it came to rewarding states that went beyond reporting graduation rates, we awarded points to those that provided information on a variety of outcomes, such as licensure passage rates, results from the National Survey of Student Engagement, and postgraduation employment rates drawn from alumni surveys. In awarding points for states that report on WA efficiency, we gave credit for an array of measures, including costs per degree, savings from operational reforms, and graduates with excess credits, among others. It is important to note that reporting on efficiency is distinct from simply reporting on revenue and costs; the key was whether the state has identified efficiency per se as something to measure and report. States could earn one point for reporting graduation rates, two for reporting an additional outcome measure, three for reporting data on two or more additional outcomes, and an additional point for reporting measures of efficiency.

We also assessed whether the state benchmarked performance in a way that would allow the public to place the data in context and gauge progress. States that used external benchmarks such as the performance of other states or the nation as a whole received the highest marks, followed by those that compared progress to state-sanctioned goals and finally those that reported year-to-year performance. On the basis of these criteria, states were graded on a five-point scale at each level (four-year and two- year), with five points signifying the best possible score.

Transparency: Consumer Information.

A second, distinct aim of transparency is to inform prospective students and families about the performance and costs of local colleges and universities. Providing consumers with better information can accomplish two goals. Individual students will have greater ability to make decisions that maximize their return on investment. At the same time, by voting with their pocketbooks, informed consumers can collectively unleash market forces that reward high- performing colleges and compel others to improve.

To score this indicator, we asked state officials whether the state had developed a mechanism to inform consumers about costs and outcomes at colleges and universities in the state. Participants were asked to include direct links to the website where these resources could be found. We were particularly interested in assessing what information was made available on student outcomes such as graduation rates, licensure passage rates, or labor market outcomes. We also rewarded states that included information about the net price of state institutions in their consumer information resources. These consumer information efforts were then graded on a three-point scale, with three points signifying the best possible score.

We believed it was important to distinguish between information that is made public in an annual report or policy brief and information that is proactively packaged and targeted to prospective students and families. To receive credit for targeting consumers, states had to place information where consumers might expect to find it—on a state website under headings such as “About Our System” or “Consumer Information,” for example.

Linking Labor Market Outcomes to Postsecondary Programs.

One of the primary purposes of higher education—though of course not the only purpose—is to train and prepare students for the workforce. Recognizing the enormous economic value of an educated citizenry, many policymakers have made a concerted push to develop longitudinal data systems to track student performance from K–12 schooling into higher education and then into the workforce.

This metric measured the extent to which states systematically tracked graduates of state postsecondary institutions into the labor market to measure employment outcomes. We made an important distinction between efforts on the part of institutions to survey recent graduates, which are relatively common, and programs that actually link so-called unit records for individual students across the postsecondary and employment data systems. While alumni surveys are useful, they suffer from serious flaws, including low response rates. We therefore focused our criteria on the latter approach. It is also important to note that we looked for states that had made these linkages and data public. Some other states may be linking postsecondary and labor market data, but unless they reported those data publicly we were not able to give them credit.

We also examined how each state reported data— specifically whether it connected labor market outcomes to individual institutions and/or to particular programs or majors, and whether it also broke down data across both programs and institutions. In other words, we gave the highest marks to those states that provided information allowing consumers to examine how the returns to a given program of study vary across institutions. We graded four- and two-year systems separately. Using these criteria, we graded states on a four-point scale, with four points signifying the best possible score.

Measuring Student Learning Outcomes.

Unlike in the K–12 world, where the standards and accountability movement has placed student learning center-stage, we have little sense of student learning outcomes in higher education. Some evidence suggests that there is ample reason for concern. In their influential book Academically Adrift: Limited Learning on College Campuses, sociologists Richard Arum and Josipa Roksa found that 45% of undergraduates had made no discernible learning gains on a standardized assessment between their freshman and sophomore years; 36% failed to do so across four years of college. Clearly, there is good reason for policymakers to pay more attention to how much students are really learning in college. Unfortunately, although a handful of national testing companies have developed postsecondary learning assessments, and some states actively use their own student assessments, in general the measurement of undergraduate learning is spotty at best.

We looked at which states are leading the way on measuring learning outcomes in higher education, and which are lagging behind. To be clear, we did not grade states on whether they had implemented assessments as a requirement for graduation or whether they mandated a particular assessment. Instead, this indicator assessed whether the state measures student learning in a systematic way across higher education institutions. We awarded additional points if they used an assessment that was comparable across states and if they made the results public. Four- and two-year systems were graded separately on a three-point scale, with three points signifying the best possible score.

Findings

When it comes to transparency and public accountability, a handful of states stood out as leaders. Texas, with its comprehensive accountability system and in-depth web-based almanac of college performance, led the charge. On each metric, Texas excelled; it has made strides to measure student learning outcomes at the University of Texas system, has linked postsecondary and labor market data, and has developed mechanisms to inform the public and policymakers about higher education performance. Minnesota and Connecticut were also ahead of most states in their commitment to transparency and public accountability. Minnesota’s annual report, Minnesota Measures, is an impressive compendium of data on many different aspects of performance, and its state university system uses an interactive dashboard to inform the public about the system’s progress.

At the back of the pack, a handful of Southern states stood out for their lack of transparency and measurement. Alabama, Mississippi, and South Carolina have not developed the means to measure learning or labor market outcomes at the college level, and their consumer information efforts, to the extent they exist, did not fulfill any of our grading criteria.

In general, transparency was lacking in the majority of states. Information on key measures of postsecondary effectiveness was remarkably difficult to obtain. While a small number of top-performers were collecting new data on student outcomes and proactively informing the public about it, most of the country has a long way to go in fostering transparency and accountability.

Detailed Findings

Transparency: Public Accountability.

We found considerable variation across the country in public reporting on the performance and productivity of state higher education systems. The vast majority of states publicly report on student outcomes such as graduation and retention rates. This is encouraging. However, most states limit this reporting to basic outcomes like degrees produced, completion rates, and information on revenue and spending.

Other states went far beyond this minimal level, reporting on student outcomes such as licensure exam results or postgraduation employment rates. At the four-year level, we found that 23 states reported data on student learning, engagement, or success on licensure exams. Twenty-three states reported labor market outcome data in some form (from either longitudinal data systems or alumni surveys). Eleven states reported some measures of both student learning and labor market outcomes. By contrast, states such as Alabama, Mississippi, and Rhode Island provided little information on student outcomes.

Just over half of the states (27) reported specific measures of efficiency and cost-effectiveness. States such as Illinois and Indiana have made it a priority to reduce the cost per degree, while Wisconsin, Washington state, and Texas have sought to reduce the number of excess credits that graduates have at graduation.

Reporting on two-year college performance was somewhat spottier. We were unable to find a public resource that provided information on student outcomes like retention or completion rates at two-year colleges in three states (Arizona, Maine, and New Hampshire). Twenty-two states provided information on student learning or licensure outcomes, and 25 reported labor market outcomes in some form. Fourteen states reported both types of data. California, Iowa, Maryland, and Texas stood out for their extremely detailed and comprehensive reports on the performance of their community colleges. Almost every state provided some kind of benchmark to place performance in context; the majority of states did more than simply provide year-over-year changes. Twenty-six states benchmarked their performance to external sources: national, state, or peer institution averages. Twelve more compared their performance in a given year with specific targets or goals set by state leaders.

A few states used multiple benchmarks, providing stakeholders with a sense of whether the state was improving over time and reaching state-sanctioned goals, and how this level of performance compared with other states or the nation as a whole. Accountability reports from the university systems in Oregon, Wisconsin, and North Carolina all provided data benchmarked to past performance, state goals, and performance in other states or at the national level. Indiana’s statewide report, Reaching Higher, also used all three benchmarks. It is important to note that a handful of states that had excellent reporting mechanisms in the past have ceased producing those resources in recent years. We considered resources from before 2008 to be out of date.

Transparency: Consumer Information.

The results for consumer information were discouraging. We found that while many states have developed glossy websites or guidebooks designed to help prospective students with college choice, the majority of these resources included no information on student outcomes. Few states included completion rates in their consumer information resources, despite the fact that these data are regularly reported to the state as part of state accountability policies. Almost no states included information on learning or labor market outcomes in their consumer resources. Only 27 states earned any points on our consumer information scoring criteria. It was particularly difficult to find quality consumer information resources for community colleges; 37 states did not fulfill any of our criteria for community colleges.

Connecticut, Texas, Minnesota, and Washington state stood out as consumer-information leaders. Washington and Minnesota have developed excellent career guidance resources that include detailed information on postsecondary programs. Washington state’s “Career Bridge” initiative allows prospective students to compare student outcomes for different degree or certificate programs (see spotlight on page 41 of the report PDF). Minnesota’s iSeek program provides a similar set of data. These websites are tailored to students in search of occupational programs, but they included data from four- year colleges as well.

These resources are a step in the right direction, but states must also recognize that all students—not only those in search of occupational programs—need better information on student outcomes at the colleges and universities in their state.

Linking Labor Market Outcomes to Postsecondary Programs.

Despite all of the action around state longitudinal data systems, we found that just 22 states made any attempt to measure labor market outcomes for their public institutions. Even fewer—a mere 12 states—connected student success in the labor market to individual institutions and/or individual programs or majors.

Connecticut boasts one of the better state efforts to measure labor market outcomes. The Connecticut Department of Education, in conjunction with the Connecticut Department of Labor, has produced a series of reports titled Building Connecticut’s Workforce. The most recent iteration (in 2010) looked at what percentage of graduates from the 18 public colleges during the 2007–2008 school year are employed in Connecticut, including by program of study. Texas’s Automated Student and Adult Learning Follow-Up System allows users to generate outcome reports for each public university and community college, as well as some private and out-of- state colleges. The reports detail employment outcomes for all graduates of the institution as well as for graduates of particular programs. These efforts are promising, though too few states have launched similar initiatives.

Measuring Student Learning Outcomes.

Even fewer states measured student learning in a systematic and comparable way. Many states simply leave it up to their institutions to decide, on a voluntary basis, if they will participate in national learning assessments; unsurprisingly, most do not. Moreover, the majority of states did not engage in their own efforts to measure student learning. Indeed, if the number of states that systematically measured student performance in the labor market was small, the number that attempt to do so for student learning outcomes is tiny. Only 13 states made any attempt to measure student learning across campuses, nine did so using a national test, and just four used a national test and clearly made the results public. Even among the four, approaches varied. South Dakota, for example, has a specific policy requiring all degree- seeking students to meet satisfactory performance requirements on the Collegiate Assessment of Academic Proficiency. In Tennessee, the state’s institutional funding model is based in part on learning outcomes, specifically on how the average institutional score compares with the national average for a given assessment. However, institutions are allowed to choose from several national tests to assess their progress, and they can apply for a waiver to test only a representative sample of their student body.

To reiterate, we are not suggesting that states should use assessments of student learning to reward or sanction campuses or as a graduation requirement. Instead, we encourage states to measure these outcomes and make the results public so that consumers and taxpayers can assess for themselves the likely return on their higher education investments.