The court's 7-1 decision Monday leaves unsettled many of the basic questions about the continued use of race as a factor in college admissions.
Justice Anthony Kennedy, writing for the court, said a federal appeals court needs to subject the University of Texas admission plan to the highest level of judicial scrutiny.
The compromise ruling throws out the decision by the New Orleans-based 5th U.S. Circuit Court of Appeals, which upheld the Texas admission plan.
Kennedy said the appeals court did not test the Texas plan under the most exacting level of judicial review.
He said such a test is required by the court's 2003 decision upholding affirmative action in higher education.
Justice Ruth Bader Ginsburg was the lone dissenter.
Justice Clarence Thomas, alone on the court, said he would have overturned the high court's 2003 ruling.
Justice Elena Kagan stayed out of the case, presumably because she had some contact with it at an earlier stage when she worked in the Justice Department.
Abigail Fisher, a white Texan, sued the university after she was denied a spot in 2008. She has since received her undergraduate degree from Louisiana State University.
The challenge to the Texas plan gained traction in part because the makeup of the court has changed since the last time the justices ruled on affirmative action in higher education in 2003. Then, Justice Sandra O'Connor wrote the majority opinion that held that colleges and universities can use race in their quest for diverse student bodies.
O'Connor retired in 2006, and her replacement, Justice Samuel Alito, has shown himself to be more skeptical of considerations of race in education.
Another factor fueling Fisher's lawsuit was that the university has produced significant diversity by automatically offering about three-quarters of its spots to graduates in the top 10 percent of their Texas high schools, under a 1990s state law signed by then-Gov. George W. Bush. The admissions program has been changed so that now only the top 8 percent gain automatic admission.
More than 8 in 10 African-American and Latino students who enrolled at the flagship campus in Austin in 2011 were automatically admitted, according to university statistics. Even among the rest, both sides acknowledge that the use of race is modest.
In all, black and Hispanic students made up more than a quarter of the incoming freshmen class. White students constituted less than half the entering class when students with Asian backgrounds and other minorities were added in.
The university said the extra measure of diversity it gets from the slots outside automatic admission is crucial because too many of its classrooms have only token minority representation, at best. At the same time, Texas argued that race is one of many factors considered and that whether race played the key role in any applicant's case was impossible to tell.
The Obama administration, 57 of the Fortune 100 companies and large numbers of public and private colleges that feared a broad ruling against affirmative action backed the Texas program. Among the benefits of affirmative action, the administration said, is that it creates a pipeline for a diverse officer corps that it called "essential to the military's operational readiness." In 2003, the court cited the importance of a similar message from military leaders.