The monumental school building on Umstead Street between Fayetteville and Roxboro streets was originally Hillside Park High School Durham's first full high school for African-Americans. The school's history began in 1887, when Durham established the first black grade school in North Carolina. It grew and expanded and, in 1922, students moved into a new building on Umstead Street next to black-only Hillside Park. By 1949, the student body and curriculum had outgrown the Umstead Street building.

The city school board decided on a swap - moving the high school to the newer J.A. Whitted Elementary building near N.C. Central University, and the elementary pupils to Umstead Street along with the Whitted name. Later, Whitted Elementary became Whitted Junior High.

The building's use as a school ended in the mid-1970s. Various organizations later used it for office space, but it has been vacant since 2005. The city, county and Durham Public Schools have discussed arrangements for repair and renovation, possibly for use as a senior citizens' center.