Streetlights and Shadows by Gary Klein, a senior scientist at Applied Research Associates, is a highly satisfying book (and there's a sample chapter on the MIT Press website). The title refers to the ancient joke: a policeman finds a drunk late at night, hunting for lost keys under a streetlamp. 'Is this where you dropped them?' the cop asks. 'No,' comes the reply, 'but this is where it's light enough to see.'

The reference is to the kinds of rules about decision making which amount to conventional wisdom – for example, teach people procedures to ensure they make better decision; collect as much information as possible before you jump to a decision; start any project with a clear description of the goal. There are situations in which the decision rules we buy into in theory do a good job. But these are straightforward, non-complex situations, which are pretty rare in life – and especially in the kinds of jobs in which they are often applied.

Take the conventional wisdom about the importance of clear procedures. This is a way of operating which has taken over the public sector. Decision-making, whether in education, healthcare, financial regulation – across the board – is codified in procedures which must be followed, and be shown to be followed. For example, banks passed their regulatory healthchecks by showing paperwork which demonstrated that they had procedures for reaching decisions about whether or not to make a loan. The wisdom of the loan was not the issue, as we have all discovered to our enormous cost. Klein demolishes, in a wonderfully low key way, the underlying idea that decisions in any but the simplest contexts can be codified in a written procedure.

Not only does the conventional model of decision making not reflect the importance of tacit knowledge, expertise gained over time, intuition, and adaptation to changing circumstances and information; it is also making for worse and worse decisions, in a vicious spiral. People don't get the sack, when things go wrong, if they followed procedures – they do if they failed to follow procedures, even if they had really thereby improved the odds of success. Even worse, doing one's best is more likely to lead to lawsuits. So everyone is demotivated from accumulating expertise and making the best decisions they could. Adequate decision-making by the book becomes the best we can hope for. And 'the book' of procedures simply gets more and more unusable as the rules are extended after every problem. Watch it happen now in financial services.

So this book is an essential read for anyone seriously interested in why government is dysfunctional. Better still, it is stuffed with practical advice for improving one's own decision-making skills, with many example's from the author's career about what approaches work, which fail, and why.

One of the most striking lessons is the need to ensure that individuals within teams who identify relevant information can bring it to the attention of the whole team, perhaps overturning a received view about the situation in question. The clearest example, perhaps, is that there were significant pieces of evidence of an imminent attack ahead of 9/11, but they were not joined together because each piece of intelligence alone was insufficient to overturn the belief that the US was unlikely to suffer an attack at home.

Other lessons include: a  mental model or theory is an essential starting point, more important than amassing plentiful information; but equally, it is essential not to fixate on a theory but to adapt it when it is countered by enough contrary information.

Ask yourself what evidence you would need to change your mind. If you can't think of any, that's a bad sign. How many pieces of evidence do you have to explain away for your theory to be true? Too many is a bad sign. What does a newcomer with a fresh eye on the situation think? Bringing in someone else is a good way to overcome entrenched group-think.

Klein's demonstration of the importance of expertise (as opposed to procedures) and analysis (as opposed to information) couldn't be more timely. The value-added in the leading economies depends more and more on tacit knowledge, as the easy-to-codify activities are outsourced either to machines or foreigners. The private sector has made a significant shift towards using new technologies in ways which make business more productive, in large part by acknowledging the expertise of employees and empowering them to make decisions. Much of private business has moved  beyond the hierarchical assembly line model.

But the public sector has hardly begun – and with the staggering pressure on public finances due to the financial crisis and the pensions crisis, it will have to make the transition quickly. This won't be at all easy, but it is terrific to have the evidence carefully accumulated and described in this book, in order to start replacing conventional wisdom about how to take decisions with good sense instead.