Drop unused to_factor levels
Removes levels that are not referenced by any value in the to_factor vector.