Authors including The English Patient writer Michael Ondaatje and Peter Carey will not attend the PEN Literary Gala in New York on 5 May.
They said PEN - known for defending imprisoned writers - was stepping beyond its traditional role.
Islamists stormed the Charlie Hebdo offices in Paris in January.
Twelve people were killed during the attack, over the magazine's depiction of the Prophet Mohammed.
Authors Rachel Kushner, Teju Cole, Taiye Selasi and Francine Prose are also against PEN handing the Freedom of Expression Courage Award to Charlie Hebdo, as they are uncomfortable with the magazine's "cultural intolerance".
Booker Prize-wining author Carey told the New York Times that the group of writers felt PEN's role was to protect freedom of expression against government oppression.
"A hideous crime was committed, but was it a freedom of speech issue for PEN America to be self-righteous about?" said Carey.
"All this is complicated by PEN's seeming blindness to the cultural arrogance of the French nation, which does not recognise its moral obligation to a large and disempowered segment of their population."
The boycott has been criticised by the author Salman Rushdie, a former president of PEN who was in hiding for years over Islamist threats in response to his novel The Satanic Verses.
He said his friends were "horribly wrong" and he hoped nobody ever came after them.
Charlie Hebdo essayist Jean-Baptiste Thoret, who escaped the attack by arriving late to work, will accept the award on behalf of his colleagues.
PEN said it did not believe Charlie Hebdo's intent was to "ostracise or insult Muslims, but rather to reject forcefully the efforts of a small minority of radical extremists to place broad categories of speech off limits."
It added: "We will be sorry not to see those who have opted out of the gala, but we respect them for their convictions."

Their lawyers had asked the court to declare "unlawful" the consultation process that led to the proposed axing of the £320m fund.
The five are among 19,000 people who receive money from the ILF, which the government plans to scrap in 2015.
The scheme aids independent living and the average pay-out is £300 a week.
The government has said that councils, which administer most social care, will take over funding this help.
On Wednesday, a judge dismissed the application, saying the consultation process had been lawful.
The claimants feared that disabled people could be forced out of independent living arrangements and into residential care, or trapped at home by the fund's closure.
At a hearing in March, they had argued that there have not been clear reasons given for closing the fund; the consultation featured inadequate information on the differences between the fund and local authority assessment and provision; and there had not been proper assessment of the impact of the change on disabled people's ability to live and work independently.
The Department for Work and Pensions has previously said that help for disabled people has been fundamentally changed by personal budgets, intended to give recipients more control, and that it makes sense to have a single system administered by local authorities.
The ILF was established in 1988, but the government decided in 2010 that it had become "no longer appropriate or sustainable" to keep running the scheme outside the mainstream social care system.
The fund closed to new applicants soon afterwards.
One of the five applicants,  Gabriel Pepper, from Walthamstow, east London, has accused the government of imposing "appalling cuts" which were "a vicious attack on the disabled".
The claimants have said the money enables them to employ personal assistants to help them with their personal needs and to "go out and have a full life".
Richard Hawkes, chief executive of the disability charity Scope, said: "Expecting councils to pick up the tab when they are facing the biggest funding cuts in history is an impossible ask."
"Not getting the support to wash, dress and leave your home is unacceptable."
Six prominent writers are boycotting a major US literary event over plans to give satirical magazine Charlie Hebdo a freedom of speech award.
[NEXT_CONCEPT]
Five disabled people have lost their High Court challenge over the government's decision to abolish the Independent Living Fund (ILF).