The shareholder revolt is another blow for the beleaguered ITV board, which is reeling after a week in which its plan to appoint former BSkyB boss Tony Ball as chief executive had to be scrapped, and executive chairman Michael Grade said he would leave the company at the end of the year, contrary to expectations that he would stay on as a non-executive.
Sir Crispin and Sir Michael Bishop, the founder of airline BMI and former chairman of Channel 4, are thought to be the front runners to take the chairman's role, and an announcement could come in the next few weeks.
However, Sir Crispin's record as chief executive of Reed Elsevier has emerged as a major stumbling block. One major ITV shareholder described his time in charge of Reed as a "disaster".
"Following a string of expensive and unnecessary acquisitions he left the business under-invested and with the highest level of debt in its history," said one top 10 shareholder.
Sir Crispin stood down as chief executive of Reed in February 2009 after almost a decade in the job.
A spokesman for ITV said: "Obviously shareholders are entitled to their own views. But the board has discussed potential candidates with shareholders who account for half the register and they are supportive of the candidates being considered."
The broadcaster abandoned the appointment of Mr Ball as chief executive last week because of his extensive list of demands, which included a pay package that, according to ITV sources, could have seen him collect £42m over five years. He also wanted the power to veto the appointment of Sir Crispin as chairman.
Mr Ball told ITV's nominations committee he "could not work with someone like Sir Crispin Davis," which was the final straw and brought an end to negotiations, according to sources at the broadcaster.
ITV's nominations committee is headed by former HBOS chief executive Sir James Crosby.
For his part, Mr Ball said ITV had issued a last-minute "ultimatum" over the appointment of a new chairman. People in his camp also said the most he would have received by way of compensation was £20m, if ITV's share price had doubled. Mr Ball's basic salary would have been about £850,000, similar to Mr Grade's.
Mr Grade left his position as chairman of the BBC to become chief executive of ITV. His tenure has been hit by declining advertising rates, exacerbated by the recession, which has led to job losses and budget cuts.
Under Sir Crispin, Reed Elsevier moved more of its operations online and charged subscription for specialist content. About 80pc of its sales come from legal and scientific research publications. He was replaced at the company's helm by Ian Smith, former chief executive of housebuilder Taylor Woodrow.
Reed issued $1.5bn (£940m) of bonds earlier this year to repay part of the debt incurred in the takeover of ChoicePoint, the insurance data and analytics company.