Now retired, Sir Robert Cooper has written the book he said he should have read before he became a diplomat

I joined the diplomatic service partly by accident.  My girlfriend, who was finishing university at the same time as me, didn’t know what to do afterwards. She was an excellent linguist who made friends easily, and I suggested she apply to the diplomatic service. When she said she couldn’t face yet another set of exams, I offered to do them too, to keep her company. I was planning to stay on at Oxford, and went into the exams in a carefree spirit. Perhaps because of that, I did well. I liked the people who interviewed me, and thought: if that’s how I look in 20 years, it wouldn’t be bad. I gave up the idea of an academic career and accepted the offer of a place in the diplomatic service – with no idea what diplomacy was about.

Nor did I learn much about it in Japan – I learned the language, loved the food and aesthetics, worked on trade promotion, and still had only a hazy notion of diplomacy.

In the 1990s, the end of the Cold War, the wars in the Gulf and the Balkans set me thinking about the strangeness and newness of our world. Out of this came a book on the changing nature of the state. Now, having retired, I have written the book I should have read before I became a diplomat: an attempt to think about diplomacy by looking at some great figures of the past.

The way to understand things is to start with facts, not theories. The military world has many books about the great battles and the great commanders. This book is about great diplomats: strategists and negotiators, and about the excitement of diplomacy, especially of the moment when you can see the possibility of a solution.

Think of Kissinger when, after several failed attempts, he and Nixon receive an answer from Beijing. They have out-manoeuvred Moscow; and they know they are going to change the world. Think of Talleyrand in Vienna, as he seduces Castlereagh, with the idea of a ‘little convention’ – in practice an alliance – that turns the tables on Russia and Prussia, cancels the looming war over Saxony by threatening a bigger war, and reclaims France’s position in Europe.

Diplomacy is where policy meets personality. Richelieu is more a strategist than a negotiator: proud, dominating, ruthless; Mazarin, who he chooses as his successor, is more human and more inventive. Richelieu had studied war; Mazarin started as a peacekeeper and a mediator.

The norm in diplomacy is failure – as those I write about illustrate. In diplomatic terms, Niccoló Machiavelli is a failure – his career finishes with Florence losing a war and its independence. But he left behind a little book that is still read today. Talleyrand spent endless time and effort trying to moderate Napoleon’s appetite for power, and failed. His attempts to knock sense into Louis XVIII were no more successful. George Kennan tried for years to get his views on the Soviet Union through to Washington, and failed for years too. Many diplomats spend as much time negotiating with their own people as with foreigners.

So add persistence, to policy and personality. The final ‘P’ is for Power. Florence failed because it was a princely state in an age when the kingly state (Philip Bobbitt’s terminology) was emerging. Richelieu’s aim was to consolidate France’s power and secure its borders. When Mazarin handed it over to Louis XIV, France was the greatest power in a Europe, and the church was no longer an organising force. At the Congress of Vienna, financial power raised its bourgeois head. When the Prussians proposed to demand massive reparations from France, Castlereagh reminded them of their even more massive debts to Britain.

The heroes of diplomacy are not found only in great powers. Small countries need diplomats too. So there is a chapter on the two diplomats who saved Denmark and Finland in their hours of crisis. Both were ruthlessly realistic; but each, crucially, had the backing of a united people. (And in Florence, it was internal division as much as diplomatic failure that brought disaster.)

No one operates alone. Sometimes it is a duo: Richelieu and Louis XIII, Kissinger and Nixon, Bahr and Brandt; diplomatic skill and political authority. Marshall and Acheson were backed by Truman. In his great crisis Kennedy brought in a range of advisers; perhaps, had they both survived, he might have found a partner in Nikita Khrushchev. In Denmark and Finland the partner was a united people –the most powerful force of all.

In the twentieth century, the end of imperialism, democracy, multilateral diplomacy and lasting alliances have transformed international affairs. Now is the time to look again at the crisis and recovery of the early post war years, at the Cuban missile crisis and the attempt at détente in the Nixon-Kissinger years. The past is past and will not be repeated; but it can help us re-imagine the future.

What do I most admire in the diplomats I write about? They are all clever; they are strong characters, who know how to persuade; and each probably has a reputation for being reliable. That much is obvious. For me, the quality that marks out the greatest is imagination. Diplomacy, like all politics, is about the future. We need to understand the reality that the past has bequeathed us; but to deal with the future needs imagination.

Today we know that the future will be about China. How should we imagine it? The answer will not come from thoughtful articles in academic journals, though they may help. It will come in action, among those who are working directly with China; from people who have the authority to make decisions and the courage to take risks.

“What is won by force is as transient as the colours of the sunset,” says Acheson. What lasts, he might have added, are negotiated solutions that have the consent of all the parties. Those who shape them in the imagination and sell them to political authority are the makers of history. If their plans do not work immediately, their ideas may yet survive. Those I write about have this sort of imagination. Attlee said of Bevin that, after Churchill, he was the most imaginative man he knew. That is why he chose him as his Foreign Secretary.

Of the talented people in this book, for me Jean Monnet is the most original, and the most imaginative. He took a problem – how to deal with the Ruhr – and turned it into a solution. And he laid the foundations for a society of states in Europe. It does not work quite as he intended – nothing ever does. But he knew that this is a world of change, where everything flows and adapts as it meets obstacles. Monnet found a solution to the problem set by Thomas Hobbes, a problem reflected in different ways by Friedrich von Genz, Metternich’s close collaborator, and by Konrad Adenauer, Germany’s great chancellor: the solitary nature of the sovereign state. Monnet did not do this alone – the one constant in diplomacy is that you need partners. He was lucky to have the support of the greatest generation of American diplomats and statesmen.

Our future depends on the kind of states we live in, and on how they live with each other. If democracy, diplomacy and imagination have taken us so far into a world of peaceful possibilities, we must believe that they can take us still further.

THE AMBASSADORS: Thinking about Diplomacy from Richelieu to Modern Times by Robert Cooper is published by Weidenfeld & Nicolson in hardback at £25, eBook £12.99