There’s a waiter in my local café who is always singing behind the counter, whistling and cracking jokes. His movements are swift, energetic. He works hard. From time to time, for no apparent reason, he winks and lets me off without paying for my tea – making me feel like a young kid and not the 38-year-old baldy that I am. Others start to be infected by his mood; his colleagues laugh and smile; customers make small talk in the queue. At first, I assume this happy vibe is something to do with the management of the place. They must be clever, cunning at incentivising their staff. Or maybe it’s the layout of the room? But then, I notice, one day when Georg isn’t there, the mood is different – less sparky, more lethargic. Is he coming back?

I start to obsess about Georg and his positive ways, wishing I could have the same fizz, the same happy energy. If only I felt that way about my work! Where does he get it from? What tricks does he have that make him so permanently upbeat? I lie in bed, staring at the ceiling as the morning sunlight crosses the plasterwork, vaguely hung over, wondering why I don’t have the same kind of spark. Does Georg not drink? Does Georg pay no attention to the morning news? After listening to the radio for ten minutes I am generally as motivated as a bull with a spear in its neck. I can hardly be bothered to move. And when I do, I ain’t happy. Finally I muster myself. A plan starts to form in my mind.

I decide to interview Georg. I decide to interview hundreds of people like him. I decide to write a book. A book! A thing I’d always dreamt of doing, and surely the solution to all of the world’s problems. Instead of hauling myself to my desk to start another day of tedious business consulting, I will become a guru, a writer. It will be my escape: a book about optimists – people who aren’t crushed by the cloud of bad news and dread that seems to hang over the rest of us; people who have that mysterious quality, the Jump out of Bed Factor.

I quit work. Over the next two years, I travel four continents and speak to dozens of perfect strangers, celebrities, psychologists and spiritual gurus. I soak up wisdom from Bill Clinton, Richard Branson, Ashley Judd. I interview David Cameron, get drunk with Harold Pinter and quarrel publicly with Desmond Tutu in a café in Cape Town. I meet a housewife who claims to be enlightened, a rabbi who surfs, and a yogi who believes he is close to attaining the secret of physical immortality.

Each one of them shares with me a different view of the universe. Each one explains why they see themselves as an optimist, what it means to them, and why I should be an “optimist” too. They grope for words, trying to capture the behaviours and outlooks that come instinctively to them, so that I can learn and pass them on in my turn.

It’s a journey that changes my life, and starts to reveal some unsettling truths. Far from being the smooth-talking wit I thought I was, I turn out to be an insecurity-riddled hustler who is terrified of failure. Why am I so scared of meeting Matthieu Ricard, a kind Buddhist monk who has been described as the happiest person in the world? Is it because he challenges my view of reality, telling me that I don’t really exist? How do I really feel about my parents? How do I feel, deep down, about myself? What am I most afraid of in the world? Those are the questions that start to define my search for optimism. The moment I start looking for happiness, it seems, I find myself face to face with my own shadow. And that’s when the fun begins.

“Don’t label anything as good or bad,” says Byron Katie, a spiritual teacher from America. “That’s why you suffer, that’s what stops you from being optimistic. Accept yourself and everything in you.”

“It’s an attitude of mind,” Richard Branson tells me, on a crackly line from his private Caribbean island. “I just enjoy every single second of my life.”

“Genuine happiness is a skill it takes time to develop,” explains Matthieu Ricard. “You don’t expect to learn chess in one day, or to ride a horse, do you? All the more so with your mind – it’s your main interface with reality.”

One thing is for sure: optimism is not what I thought it was. Rather than being dependent on the outlook for the global economy or the tone of the Today programme, it seems to be a thing of the mind, an attitude.

But what kind of attitude? Is it just about looking on the bright side? Or is there something more to it? I discover that the word itself is loaded with ambiguity. Only recently, it turns out, did optimism come to be associated with looking forward to a better future. Coined by the German philosopher G. W. Leibniz in the 18th century, optimism was originally intended to describe his theory that this was the best possible universe out of an infinite number of possible universes. By this definition, optimism is nothing to do with the future, it is to do with now, and the essential perfection of now, warts and all – including all the dark and painful bits we would rather not know about. As Ashley Judd puts it when I interview her on the phone from her ranch in Tennessee, “The best exists inside of me as well as the worst.” It is an idea of positivity we have completely forgotten in our obsession with progress and technology, and our natural urge to eradicate suffering. And yet it is an idea – I realise – that fits much better with our modern times, with its troubles, uncertainty and its New Age aspiration of living “in the present”.

The next thing I learn is that anyone can be an optimist. According to Martin Seligman, an American psychologist and a pioneer of the clinical study of positive thinking, there is no genetic limit to our ability to find happiness. Rather, optimism – and pessimism – are tendencies, sets of learnt reactions that govern how we respond to adverse experiences. Sure, some people are born with a tendency to focus on the bright side, but – as Seligman has proved – it can equally well be cultivated. And as we pace around York Minster on a cold autumn afternoon, the grey-haired professor explains his secrets to me and gives me his blessing to pass them on to others.

I start to see a common thread in my encounters. Optimism is not about the future and it’s not about the news: it’s in my power. It’s about how I react to events, the moment they occur. It’s in the mind. The real optimists, the people I admire the most, have a set of beliefs and routines for reacting that keeps them happy and alive.

And yet, after two years of interviews, I still find myself waking in a fog. I still stew in envy when I think about Richard Branson, who tells me that he really does spring, freshly minted, from his bed every day at 6am. Or Lynn, a multiple sclerosis patient who confides that she loves every minute of her life. Meanwhile, I still wake with a cloud over my head and the vague feeling that there is something amiss.

The missing piece in the puzzle comes when I travel to Johannesburg to meet Taddy Blecher, a young South African who left his job as a highly paid consultant to start a business school for underprivileged kids. The man’s undistilled confidence, his absolute optimism about humanity and its potential, are so inspiring that I can feel my skin literally fizzing as I listen… That’s what I am looking for. That’s what these optimists have in common. A sense of purpose. A meaning bigger than ourselves and our own little concerns.

I return to London determined to remember Taddy’s words about optimism and the rush of excitement that comes with believing in the world. I start to feel a sense of possibility about my future, something I don’t remember for years, possibly since my childhood. But it is only when I listen to Bill Clinton at a crowded fundraiser in Greenwich that it all comes together. “Think about it!” he says. “We organise our minds to obsess about things that don’t amount to a hill of beans. You be free now!”

Today, I still consult for companies; today, I still stare at the ceiling as I blink myself awake, wondering where to get the energy I need for my day. But now – three years after I first had the idea for the book – optimism has become something very simple for me. It’s like Bill said: I can make a choice. Indulge the worries of my obsessive mind, or simply let them go. Now I make that choice every day of my life.

I’m still curious about Georg, though. When I finally get around to asking him what he thinks about it all, as I queue for my tea, there is a twinkle in his eye. “You just stay happy,” he smiles, spinning a china plate. “And then you make everyone else happy too!”