Spies! Secrets! There is probably more rubbish written about these subjects than about anything else. Why? Because governments normally neither confirm nor deny reports concerning alleged espionage, so the world’s avid conspiracy theorists write what they like without fear of official contradiction.

All of which suits intelligence agencies just fine. Media hubbub about double-crossing, fancy gadgets and sultry temptresses helpfully draws public attention away from the hard-edged reality of what they do.

Spying has a long history. People always have had plans and information which they wish to keep to themselves, which of course encourages others to make it their business to find out what is ‘really’ going on. This has created an endlessly inventive industry of secret codes and procedures and special equipment intended to keep secrets secret, matched by no less inventive techniques aimed at unobtrusively penetrating those defences.

The problems would-be spies encounter have not changed much over the centuries. They are something like this:

• identifying where sensitive and useful information might be stored or circulated;

• identifying weaknesses in its protection (human or technical/physical);

• using those weaknesses to access the information;

• copying it in an undetectable way;

• getting that information back to HQ;

• if at all possible without anyone noticing or suspecting and

• preferably many times over – a steady flow of good information is much more useful than a one-off.

These days the most intense espionage effort is the silent arms-race between rival national computer programmers. The FCO and MI6 are under cyber-bombardment from hostile intelligence services and opportunist hackers. Computer attacks look for electronic loopholes in systems to suck out information.

Even unclassified information can be useful once aggregated and crunched through powerful computers. HR data or even patterns of telephone calls help narrow down who is doing what job – or not.

Machines can do amazing things. But not everything. The main vulnerability in any system is people. Unreliable and unpredictable. They get tired and make mistakes. They may not be as clever as they think they are. Above all, someone working ‘for the other side’ is under huge pressure, as the penalties for treason are usually severe. But if one side can persuade someone on the other side to betray their organisation, colleagues and country, wonderful grazing opens up for spying.

One of the most spectacularly successful early spymasters was Sir Thomas Walsingham, English Ambassador to France from 1570-73 before he set up ingenious schemes to thwart Catholic plots against Queen Elizabeth 1. He secretly followed the infamous Babington Plot for months, before Mary Queen of Scots unwisely wrote a letter supporting the plotters which was intercepted and copied by Walsingham’s team. Mary was beheaded and numerous plotters were hung, drawn and quartered (ie disembowelled alive) as a mark of the severity of their crime.

One of the last victims of this most cruel of sentences was a French spy, Francois Henri de la Motte, who had sent Paris important information on English naval movements. His vivid execution in 1781 attracted an  enthusiastic crowd of some 80,000. A year later came the last such execution, of Scotsman David Tyrie, working for Paris. Nowadays we have the European Union’s Common Security and Defence Policy, so spies die of boredom instead.

These historic examples illustrate the main difficulty facing someone within state A, who wants to send its secrets to state B, undetected. As it happens, I was a student at The Fletcher School of Law and Diplomacy in the 1970s with a young American who went on to achieve utter ignominy as an agent working for Israel deep in US Navy structures. Jay Pollard betrayed his country on a prodigious scale, sending tens of thousands of top secret documents to the Israelis. He was given a life prison sentence in 1987 but is expected to be released in 2015. Israel has made representations for his release at the highest level in Washington. Successive US presidents have taken no notice, a sure sign that Pollard’s perfidy was on an epic scale.

The Russians traditionally have given much thought to espionage technique and have had a generous share of successes and failures. I played a walk-on part in one of their supposed successes back in 1996. British ambassador Sir Andrew Wood and I were abruptly summoned to the Russian Foreign Ministry in Moscow one bright Monday morning and told that a sizeable group of British diplomats had to leave within two weeks because of ‘activities incompatible with their diplomatic status.’

This was the first serious spy scandal after the end of the Cold War. London, of course, retaliated by expelling Russian diplomats. It all petered out. More recently, in 2006, the Russians showed TV footage of British diplomats apparently collecting information from an electronic device hidden in a false rock. Showing impressive new technique, the Russian government decided not to expel the diplomats concerned, but tittered at them publicly instead.

Yet these victories (if that is what they were) for the Russian side were as nothing compared to the recent ghastly debacle the Russians suffered when their invaluable network of ‘sleeper’ agents was rolled up by the FBI in the United States.

Again, much media coverage was nonsensical or dishonest, playing down the significance of these agents: ‘Ho Ho, they must have been really dangerous, spying on their local schools!’

It is not easy for (say) a Russian diplomat or ‘businessman’ repeatedly to approach a US official or to sniff around a US government agency without raising some suspicions. So, why not use Russians who look like Americans?

To carry this off requires years of unproductive work as false identities are painstakingly constructed and the would-be Americans try to manoeuvre themselves into useful places.

Some might hit the jackpot and get a job in a sensitive facility, or marry someone whose close friend is in a sensitive facility, or be part of a local community where people who work in sensitive facilities hang out.

Information isn’t everything. The sleepers may do well by helping spot weaknesses (‘Joe Jones’s cousin Fred works in the Pentagon Telecommunications Centre and is having a messy divorce and drinking problems. Mary Kennick’s husband knows someone in the White House protocol team who has heavy gambling debts…’)

Plus, think about the problem of conveying stolen information safely to Moscow. It could be dangerous for a Russian mole in the US Air Force to pass information regularly to Russian diplomats – far easier to swing through a leafy suburb and slip it to an unsuspected sleeper.

Or consider the difficulty in getting microphones planted in a building. Not so easy for a Russian to run a recce and see how the security works. But an ‘American’ sleeper might be able to do that for you.

In short, without looking carefully at the whole production chain of intelligence information, it makes no sense to snigger, as so many Western media outlets disgracefully did, at the significance of this excellent US power-play against this laboriously-established and valuable Russian network.

To make matters even worse for Moscow, it looks as if the Americans had been following this group for years, learning all sorts of new things about Russian top-end technique in the process. So why arrest the sleeper group now? Because the whole operation was maturing on schedule, ie they were getting too close to senior Americans?

All of which explains why Moscow was so keen to set up a rapid ‘spy swap’ and get these now wide-awake sleepers back to Mother Russia as fast as possible, to minimise the embarrassment but much more importantly to find out just what damage had been done.

One aspect of espionage which has changed in the past few years has been the new willingness of intelligence agencies to offer their face openly to the general public. Here in the UK  MI5, MI6 and GCHQ each have a cheerful website describing their noble missions and even including careers information. Back in Moscow Russia’s Foreign Intelligence Service (SVR) has much the same.

Such agencies have to decide what information to share with the public, and how to do it. The MI6 website has the droll touch of Language Options to make life easier for hostile intelligence operatives: Spanish, Russian, French, Arabic and Chinese. Do we really need to know that GCHQ staff are running a Christmas charity campaign called ‘Pants for Christmas’, collecting new underwear, toiletries and other clothing for homeless people?

Let’s leave the final thought with John Le Carré:

A spy, like a writer, lives outside the mainstream population.
He steals his experience through bribes and reconstructs it.

PS:  Just to add that I have dictated this article straight into the computer using snappy voice recognition software which has come on leaps and bounds since I tried it out back in 1999 for the FCO.

Why is such brilliant technology not used by every Foreign Ministry? Because a hostile intelligence service might penetrate the network, find a computer using voice recognition, activate the microphone by remote control from thousands of miles away, and sit back happily listening live to every conversation. Nice.

French spy, Francois Henri de la Motte’s vivid execution in 1781 attracted an  enthusiastic crowd of some 80,000…Nowadays we have the European Union’s Common Security and Defence Policy, so spies die of boredom instead.