Rupert Thomson is the author of nine novels, including The Insult (1996), which David Bowie chose for one of his 100 must-read books of all time, and Death of a Murderer, shortlisted for the Costa Novel of the Year awards in 2007. His most recent novel, Secrecy, was hailed as "chillingly brilliant" (Financial Times) and "bewitching" (Daily Mail). According to the Independent, "No one else writes quite like this in Britain today." Thomson has also been compared to JG Ballard, Elmore Leonard, Mervyn Peake and even Kafka. In short, he's an established and successful writer with an impressive body of work to his name.

After working seven days a week without holidays, and now approaching 60, Thomson, you might think, must be looking forward to a measure of comfort and security as the shadows of old age crowd in. But no. For some years he has rented an office in Black Prince Road, on London's South Bank, and commuted to work. Now this studio life, so essential to his work, is under threat. Lately, having done his sums and calculated his likely earnings for the coming year, he has commissioned a builder to create a tiny office (4ft 9in x 9ft 11in) at home in his attic, what he calls "my garret".

The space is so cramped that Thomson, who is just over 6ft, will only be able to stand upright in the doorway, but he seems to derive a certain grim satisfaction from confronting his predicament. "All I want is enough money to carry on writing full time. And it's not a huge amount of money. I suppose you could say that I've been lucky to survive as long as I have, to develop a certain way of working. Sadly, longevity is no longer a sign of staying power."

Thomson is not yet broke, but he's up against it. The story of his garret is a parable of literary life in Britain today. Ever since the credit crunch of 2008 writers have been tightening belts, cutting back and, in extreme cases, staring into an abyss of penury. "Last year," said novelist Paul Bailey, speaking to the Observer in 2010, "was sheer hell". Off the record, other writers will freely confide their fears for the future, wondering aloud about how they will make ends meet. Hanif Kureishi, for instance, recently swindled out of his life savings, told me how difficult his life had become. Never mind the money, the very business of authorship is now at stake.

After a period of prosperity and tranquillity for British fiction that ran for about a generation (circa 1980 to 2007), writers are now being confronted with the hardship of literary artists through the ages. (It was said of Grub Street's 18th-century residents that "They knew luxury, and they knew beggary, but they never knew comfort.") Thomson, reviewing his situation and hoping, like Mr Micawber, that something turns up, says: "I don't buy anything. No clothes, no luxuries, nothing. I have no private income, no rich wife, no inheritance, no pension. I have nothing to look forward to. There's no safety net at all." He, and many others, are having to face up to unprecedented questions about their survival as writers.

In retrospect, the turning-point in British writers' fortunes came in 1980. The Booker prize was televised for the first time, and the subsequent year Midnight's Children won. After that, literary life began booming – a mirror to the irrational exuberance of the economy. Tim Waterstone's bookselling revolution was transforming the trade. New writers were making headlines. In the feeding frenzy that followed, publishers' advances entered a never-never land in which commercial prudence was thrown to the winds.

Thomson remembers exactly when this party came to a stop. He was at a publishing do in the autumn of 2008 and had fallen into conversation with Lee Braxton, an editor at Faber & Faber. "He told me," he recalls, "and not for the first time, how much Faber would like to publish my work. And then he said, 'But I can't afford you.' So I asked him what he would pay, and he named a figure for a two-book deal. That was the first time I noticed the drop in advances because the figure that he gave was only a fraction of what I'd been getting up to then. I went home and sat at the kitchen table and drew up a balance-sheet. I thought: I'm going to have to change the way I live."

Thomson is a veteran from a now-deserted battlefield. Rates of attrition among so-called "mid-list" writers, steady professionals who can no longer find publishers to support them, have begun to rise alarmingly. But drop a generation or two, and you find parallel stories: young writers grappling with a wholly new – and in some respects, hostile – literary landscape. In a business that relies on keeping up appearances, no one wants to admit this. Privately, there's a lot of fear.

‘I have no private income, no rich wife, no inheritance, no pension. There’s no safety net at all’: Rupert Thomson' Photograph: Richard Saker for the Observer

Money, once plentiful, has become suddenly scarce. Consider the story of Joanna Kavenna, a former Observer critic, acclaimed author of The Ice Museum (2002) and winner of the 2008 Orange Award for New Writing. Kavenna, who was selected in 2013 for Granta's Best of Young British Novelists promotion, recalls her beginnings as an author and novelist, switching countries in search of cheap rent and the kind of job that would give her time to write. "I was living in Oslo when I got the idea for The Ice Museum, a non-fiction travelogue about the remote North. I was offered a – to me, miraculous – five-figure deal by Penguin." While researching and writing the book, Kavenna was advanced, in instalments, and before tax or her agent's commission, a sum of roughly £10,000 a year.

So far, so familiar. The publishing boom was still in full swing (just). Looking back, Kavenna remembers the experience of writing and publishing The Ice Museum with some affection. "My initial career conformed to my original notion of authorship," she says. "You wrote a book, you struggled to get it published, if you were lucky you found a kindly editor who paid you a bit of money, and later perhaps you'd be paid for another book. And so on. In the meantime you did other things like secretary work or journalism to make a bit more. But then, between 2007 and 2010, everything changed."

Here, Kavenna reels off that catalogue of woes commonly shared among writers today: book review sections cutting back; publishing houses worrying about the future; marketing types calling the shots; libraries closing; bookshops going out of business; the dread march of Amazon. Like many in this community, she also worries about the surge in social media, the rise of Facebook, Twitter and the blogosphere, ie internet sites where anyone can put up "free content", either for pleasure or self-promotion, or from a confused mixture of both instincts. Put these anxieties together and you have a picture of a way of life facing extinction. In summary, she says, "being a writer stopped being the way it had been for ages – the way I expected it to be – and became something different."

That "difference" amounts to a revolution. To writers of my generation, who grew up in the age of Penguin books, vinyl records and the BBC, it's as if a cultural ecology has been wiped out. For as long as most of us can remember, every would-be writer knew the landscape of the printed word. This Georgian square was home to publishing grandees (now retired). On that high street were the booksellers (now out of business). In those twisting back streets, you could expect to find literary agents working the margins with the injured innocence of pickpockets at a synod. It was a mutually dependent ecosystem.

Publishers were toffs, booksellers trade and printers the artisan champions of liberty. Like the class system, we thought, nothing would change. The most urgent deadline was lunch. How wrong we were. The years 2007-2010 are pivotal: first, as Thomson has described, came the credit crunch. And it occurred at the very moment that the IT revolution was wrecking the livelihoods of those creative classes – film-makers, musicians and writers of all sorts – who had previously lived on their copyrights.

Roughly speaking, until 2000, if you wrote a story, made a film or recorded a song, and people paid to buy it, in the form of a book, a DVD or a CD, you received a measurable reward for your creativity. Customers paid because they were happy to honour your creative copyright. When the internet began in the 1990s, many utopian dreams of creating an open society, where information would be free for all, sprang into prominence. Wikipedia, for instance, is the child of such dreams. Today, Wikipedia is appealing to its users for subscriptions.

Among many champions of the open (and free) society, Jaron Lanier, author of You Are Not a Gadget and Who Owns the Future?, celebrated the idea of knowledge without frontiers from the comfortable security of a university post. The reckoning has been slow in coming, but now there are some crucial indicators of a change of heart. Lanier, for example, acknowledges that, in his excitement at the birth of the worldwide web, he forgot about the creative classes. He concedes that he has watched a generation of his friends – film-makers, writers, musicians – become professionally annihilated by the loss of creative copyright.

Copyright is the bone-marrow of the western intellectual tradition. Until the book world, like the music world, can reconcile the extraordinary opportunities provided by the web with the need for a well-regulated copyright system, artists of all kinds will struggle. A bleak situation sponsors desperate remedies. Thomson confides that, "if someone said to me, 'You must give up two of your [already published] books to publish two more new ones,' I would agree to that."

Kavenna speaks passionately about the copyright problem from the perspective of a young writer struggling to protect her livelihood. Recalling, with horror, the so-called Google Print Initiative (the digitising of the world's copyright libraries), she says: "It's as if you came home and found your house burgled, but when the police turned up they said: 'Did you stick a sign outside your front door saying: DON'T BURGLE MY HOUSE?'"

For writers in Britain today, the future is incredibly uncertain for all but the mega- selling super-authors. This, in turn, jeopardises the culture as a whole, threatening a world in which a handful of prizewinners dominate the market. Rupert Thomson, who has had more than his fair share of acclaim, says: "A prize is worth a thousand rave reviews. Prizes have become essential, but it's no longer about glory, it's about survival." The Royal Literary Fund, for instance, exists to help writers in difficulty. One of its members told me that "Whenever we meet now, half the committee are thinking: 'There but for the grace of God…'"

Despite the uncertainties, new writers continue to struggle into the light. Tara Isabella Burton, from New York City, is one of Kavenna's protégés, a writer and theology student who has just submitted her novel, A Thief in the Night, to her agent. Burton, who was born in 1990, is a child of the digital age. She has never known a world without computers and says she has "wanted to write since childhood". At first, she made ends meet as a freelance ghost writer, supplementing this with bits of travel writing and book reviews for the Spectator, the New Statesman and the Atlantic.

"There's a lot of waiting to hear, and waiting to be paid," Burton says, philosophically. "I'm much more comfortable now about negotiating for money up front." Burton insists on being paid for her writing: "It's a bad precedent to say, 'It's for the love.' And it's hard to take the work seriously if it's for free." So what about her future ? "I'd like to keep on doing everything, but I can't afford to write for free. At least with the internet, I don't feel constrained by time or place. Sometimes I've been up a mountain in Georgia while writing for a magazine in Los Angeles."

For Kavenna, this freedom is a reason to be optimistic about the future: "The digital age," she says, "is an extraordinary revolution in consciousness. I grew up with the Modernists – Joyce et al – grappling with the technological developments of the early 20th century. The digital age is just as significant. We are developing a completely different mode of consciousness. So the digital age offers this new challenge for writers."

Kavenna also recognises some exciting new opportunities: literary magazines, such as Spolia, which pool its revenues among their contributors, and Arc, which publishes a super-tech online version – followed by an aesthetically pleasing physical version. These, of course, are just formats. Creativity begins with a genuine compulsion, an active desire to write, compose, etc. And what follows is the question of how society rates the artist. Does society regard such enterprises as important and worthy of status, or, as with hyper-capitalism, something that should live or die by the market and nothing else? It's at this point that the darkest anxieties of the night crowd in.

Rupert Thomson outside the office he rents on London's South Bank Photograph: Richard Saker for the Observer

Rupert Thomson, for instance, admits that his worst fear is that "I have to stop writing. That's something I've had to face in 2013." Then he perks up. There's his garret to finish, and more things to commit to paper. "I can't really imagine a life where I'm not writing. I've got this ludicrous faith that I'll be able to go on as I am now. That's all I want." For a moment, Thomson might almost find himself echoing Mr Micawber: "Welcome poverty! Welcome misery, welcome houselessness, welcome hunger, rags, tempest, and beggary! Mutual confidence will sustain us to the end!"

Kavenna finds a less romantic inspiration for beating on, upstream. "I get the impression that people are sick of being lied to by corporations and governments," she says. "Writers have nothing but their integrity. They are disaffiliated. They can tell the truth. Anything doing that might just get an audience, whether it comes as a physical artefact like a hardback, or as an e-book. You don't write unless you hope for that. And you always hope."

• This article was amended on 27 February 2015 to change a picture and further amended on 1 July 2015 to clarify the publisher's advance figure for The Ice Museum.