Why Incomplete Data Leaves Us With Incomplete Student Outcomes

by: Amanda Janice Roberson

How many times have we heard that graduation rates count just a fraction of current college students because data focus on only those who are first-time, full-time (FTFT)? Given that only 47% of students are FTFT, our data are not representative of today’s students and, frankly, we cannot afford to keep making these mistakes.[1]

In recent years, the U.S. Department of Education (ED) has spearheaded efforts to improve the data available to consumers and policymakers by releasing the revamped College Scorecard in 2015, which included data on employment and loan repayment. However, the data pitfalls are shockingly similar to graduation rates: some of these data, like employment and earnings, are limited to only those students who receive Title IV federal student aid, omitting 30% of students.[2]

As a result, data are not representative of significant portions of the student body and are insufficient for consumer information and policymaking purposes. National postsecondary statistics and an improved federal data system, like a student-level data network, must count all students, whether or not they receive federal student aid.[3] Here’s why:

All students deserve accurate, representative information, regardless of whether they receive federal student aid.

Students need quality information to make college choices. If a student does not receive federal aid, we should not stand in their way of accessing representative information about outcomes at individual colleges and programs. States and institutions cannot provide this information reliably, completely, and in a comparable manner, on their own because only the federal government has access to complete earnings information. Simply stated, we will not have a clear picture of higher education in the 21^st century until data systems are comprehensive and represent all students and all outcomes. All students deserve to know what they can expect in return for their investment of time and money.

Without all students, a student-level data network cannot replace Integrated Postsecondary Education Data System (IPEDS) reporting.

One promising feature of an SLDN is its ability to replace the student components of IPEDS. Institutions must calculate upwards of 500 metrics, including, but not limited to, enrollment, completion, and pricing metrics, disaggregated by a variety of student characteristics. Aggregate IPEDS data include students who do and do not receive federal student aid so an SLDN must include aided and non-aided students to replace IPEDS and streamline institutional reporting.

Federal investment in postsecondary education benefits all institutions and all students, not only those who participate in federal grant and loan programs.

The federal government subsidizes institutions and students with a more than $162 billion annual investment in higher education through Title IV programs.[4] It also supports institutions of higher education through research grants from the National Institutes of Health and National Science Foundation, among other agencies, as well as through Title III and Title V programs, which authorize funding and support for Minority Serving Institutions. Federal support outside of the Title IV programs enables institutions to operate, benefitting students beyond those who directly receive federal grants and loans.

Students benefit from further federal investment like the American Opportunity Tax Credit and other exemptions and deductions, which provide tax relief for students and their families while attending college, and GI Bill benefits and the Tuition Assistance Program, which provide financial assistance to servicemembers and veterans. If federal data systems exclude students who do not receive federal student aid, the data will not reflect the scope of outcomes for students who benefit from other federal investment.

Federal data would not accurately reflect institutional performance because significant proportions of students do not receive federal student aid.

The omission of students who do not receive federal student aid from performance metrics would skew results dramatically for some institutions. Only about 62% of community college students receive federal financial aid, compared with 70% of all undergraduates.[5] If a data system omits non-Title IV recipients, then more than one in three community college students will be excluded from postsecondary performance metrics. Some colleges and systems would be unduly mischaracterized by these incomplete data: in 2016-17, only 18.4% of California Community College System students received Pell Grants and 1.5% of students received some type of loan. When we count only federally aided students, we hide the outcomes for about four in five enrolled California community college students.[6]

Similar concerns already exist about federal graduation rates, which only count FTFT students. The field argues that these rates are misleading and it behooves the higher education community to require representative data from institutions.[7] A higher education marketplace cannot function in the absence of equal information—both students and institutions deserve information that reflects the full student body.

The federal government already collects some information on non-federally aided students.

There is precedent for the federal government to collect data on non-aided students. For example, each student receives a 1098-T form from their university during tax season that includes tuition and scholarship amounts for the tax year, regardless of their federal aid status. Institutions submit these forms with student-level information to the Internal Revenue Service to aid in the administration of education tax benefits.

Additionally, to be listed as an Eligible Training Provider under the Workforce Innovation and Opportunity Act (WIOA), an institution must provide program-level data. This requires the collection and reporting of outcomes for all students, including those who do not receive WIOA funding. WIOA and 1098-T reporting show clear precedent for the federal government collecting data on non-aided students to provide value back to students and families, institutions, and policymakers through better and more complete information.

To be champions of equity and civil rights, we need data that allows policymakers and institutions to identify and close socioeconomic gaps.

Omitting non-Title IV students limits our collective ability to understand equity in higher education and provide students with the appropriate information to make informed decisions. Just as the Every Student Succeeds Act requires disaggregated data to be reported on the performance of economically disadvantaged students as compared to students who are not economically disadvantaged, we need high-quality information on low-income students (i.e., Pell Grant recipients) and non-low-income students (i.e., students who do not receive federal aid). Without complete, representative data, equity will be out of reach and policymakers and institutions will not be able to enact evidence-based policies that promote college access, success, and affordability for all students.

The solution is clear: in order to reflect the outcomes of all students, we must include non-federally aided students in federal data systems. The ability to advocate for students, reflect institutional performance, and streamline reporting depends on it.