They chose retirement after the head of the Department of Corrections and Rehabilitation "determined that change in leadership at those prisons is necessary" following recent reviews that identified problems, spokesman Jeffrey Callison told The Associated Press in an email.
He did not detail the problems but said the department is committed to making necessary changes at the prisons "to comport with our vision of safe, rehabilitative environments for inmates."
State corrections officials earlier had called the retirements routine.
Johnson retired after 30 years of state service and Hughes after more than 27 years, officials said.
The department said it could not arrange interviews with the retired wardens and The Associated Press could not immediately locate them. Two other senior staff members at the Chowchilla prison also were being reassigned.
Specter pushed for leadership changes after attorneys found systemic problems at the Chowchilla facility, which he called "a very troubled place."
"There are serious problems there, including verbal abuse of prisoners, failure to protect them from other prisoners, contraband, sexual abuse -- mostly in the form of: 'If you do me a favor, I'll do you a favor' -- that kind of thing," he said. "There's a lot of fear, and fear of retaliation for reporting misconduct."
Inmates allege that guards "will use their authority to coerce prisoners into sexual relationships," Specter said. "Some of the women allege that the officers bring in contraband, partly in return for sexual favors."
Some guards have let inmates from one housing unit into another to fight or steal property, he said.
"They ignore fights, they let them go on even though they know about them, they ignore pleas for protection," Specter said.
Guards use unnecessary physical force, and "there's a rampant culture of verbal abuse," that includes guards using demeaning names for women.
The attorneys, who represent inmates in several major lawsuits against the state, have been working with the corrections department in recent months, including conducting joint interviews in June with about 150 of the more than 2,800 inmates at the Chowchilla prison, he said.
Specter said the department appears to have been taking steps to make changes since then.
Nichol Gomez, spokeswoman for the California Correctional Peace Officers Association, said in an email that the union that represents most guards "does not comment on unsubstantiated rumors or allegations."
"But I can tell you the majority of California correctional peace officers are professional and take their duty and oath seriously," she wrote. Employees at both prisons "do their job the best they can within the conditions they face, which include lack of staff."
Specter praised the leadership change but said attorneys are waiting to see what else the department will do to change the prison's culture and practices. He equated problems there to a December finding by the state inspector general saying High Desert State men's prison had a culture of racism and abuse.
Meanwhile, Democratic state Sen. Connie Leyva wants the state auditor to look into suicides at the California Institution for Women, which houses nearly 1,900 inmates in Corona.
She is asking for an audit next week to see why the suicide rate was eight times the national average for female prisoners in an 18-month period in 2014-15, when The Associated Press first wrote about the spike. There were four suicides and at least 35 attempts during that period, and Leyva cited two more suicides this year.