The Ministry of Justice could face legal action over the ban on sending books to prisoners.
Eminent human rights lawyer Geoffrey Robertson QC said that the Lord Chancellor, Chris Grayling, could be taken to court for acting "unlawfully and irrationally".
It follows the launch of a major campaign by the Howard League for Penal Reform, supported by English PEN, calling on the government to end restrictions which prevent families and friends sending books, underwear and other essentials to prisoners.
Tens of thousands of people have shown their support for the campaign by signing a petition and sending photographs of bookshelves to the Ministry of Justice's Twitter account using the hashtags "#shelfie" and "#booksforprisoners".
Mr Robertson, Joint Head of Doughty Street Chambers, the largest human rights practice in Britain, said that, if the restrictions that affect the sending of books to prisoners are not lifted, the Ministry of Justice could be taken to court for acting unlawfully and irrationally and for denying prisoners their right to receive information.
He said that it might additionally be argued that Mr Grayling was breaching the 1688 Bill of Rights by taking it upon himself – as a politician and not a judge – to inflict "cruel and unusual punishment" on prisoners who are literate.
Mr Robertson added that the new restrictions appeared to involve religious discrimination because "the only books that will not be turned away under this scheme are the Bible and other religious texts".
Geoffrey Robertson QC, Joint Head of Doughty Street Chambers, said:
"Mr Grayling is not a lawyer; he is a politician who seems to think he is above the law. He has no power to impose additional punishment on prisoners over and above that which is imposed by the courts. The action has nothing to do with prison security or any other legitimate purpose.
"The right to read is precious in this country and for prisoners it is a way to lift themselves out of the slough of criminality. To deny them the books they need in order to improve themselves is both unreasonable and counter-productive."
Frances Crook, Chief Executive of the Howard League for Penal Reform, said:
"The Ministry of Justice should recognise this policy is flawed and the ban on loved ones sending in books and other essentials to prisoners has little or no support from anyone with an ounce of common sense.
"Reading books goes hand in hand with education, with rehabilitation, with humanity. The spurious arguments in favour of this measure simply don't stand up to scrutiny.
"With ministers digging their heels in despite the fantastic expressions of support we have received from famous authors and ordinary members of the public, we are now with regret considering legal action. We still hope that the Ministry of Justice will see sense and work with us to resolve the unintended consequences of this policy."
Jo Glanville, Director, English PEN said:
"Through our work in prisons, we've seen at first hand the necessity of access to learning and literature. The government needs to review its short-sighted policy urgently.
"This is a counter-productive measure that has shocked the public and the literary establishment, and we're very disappointed by ministers' response to the campaign.
"We should not need to resort to legal action to convince the government to drop such a manifestly unpopular and ill-considered policy."