It used to be the case that pastry was the least respected station in the professional kitchen.

Meat, fish, sauce and larder were deemed much more important. All that has changed and pastry chefs, with their very specialist precision skills in sugar, chocolate, fruit and cake - hot, warm, chilled or frozen - have become so revered that it's almost de rigueur for any head chef to have come through the rigours of pastry. You only have to glance at a dessert menu to see how the discipline has re-asserted itself as an equal - if not leading - partner in the fine-dining restaurant offer.

Much of this is driven by demand from traditionally sweet-toothed diners, of course, but other more recent factors are involved too. One-upmanship between the chefs themselves, whose access to the latest technological gizmos drive them towards ever foamier espuma. Then there's the competitive telly programmes like MasterChef and the Great British Bake Off, where puddings are judged as harshly as mains. Add a constant stream of glossy books and features by a new wave of progressive patissiers and you have a pudding-positive public whose expectations are as high as a passion fruit souffle.

Inevitably this has had a trickle-down effect on retail, where desserts is the most intensely competitive area in supermarkets. I got a glimpse into how dog-eat-dog the dessert market is when I met with James Campbell, the Scottish head of dessert development at M&S for the last two years. With a remit to overhaul the entire chilled (as opposed to frozen) range he has already launched or rebranded over 50 products to stay ahead of the game, and describes his job as constantly innovating and staying abreast of global trends while trying to appease the traditionalists. Or, as he puts it, "Finding the balance between nostalgia and being genuinely first-to-market."

His new British raspberry and vanilla rose dessert, complete with the molecular, El Bulli-inspired, self-saucing yolk, glitter spray glaze and exquisite aesthetic appeal, took 18 months to perfect and is one example of the latter. Entirely handmade, it contains no fake flavourings or preservatives and uses only natural colourings; sugar content is low.

Like his "hero signature product" the Jaffa Sphere, 4000 of which are made every day to keep up with demand, it sells out regularly. It's all part of Campbell's quest for what he calls a "trend-driven Michelin star restaurant quality" desserts range. Such innovation - much of it acquired from culinary trips around the world - is aimed at attracting the new demographic of younger sweet-savvy customers, but it's also vital to keep the older ones satisfied too: one change too many and they are lost forever.

I'm told there are certain items, such as apple crumble and sponge puddings, that can't ever be taken off the shelves, and it seems M&S's legions of loyal customers can tell instantly if defatted cocoa has been introduced to a favourite dessert.

On the other hand, seaweed, brown algae, tonka bean and green tea are unavoidably on-trend. Keeping this balance is one major challenge when your customer base is spread across the UK and beyond, and when each product has to be reproduced to the same consistency some 5000 times per day.

It's significant that Campbell, 40, was headhunted for the job from the high-end restaurant business: a classically French trained chef, he had worked with Gary Rhodes and at age 23 was head pastry chef at the Michelin-starred Rhodes in the Square in London's Mayfair, later becoming consultant chef to the Rhodes group. He then worked at the acclaimed Mandarin Oriental in Knightsbridge, in charge of a team of 12 pastry chefs. Not bad, given he was born in Bonhill, in the Vale of Leven, and left Our Lady and St Patrick's high school in Dumbarton with seven O Grades intending to study accountancy at Clydebank College.

First he needed a job, so he rang Cameron House Hotel at Loch Lomond to be told they were looking for a commis chef and a leisure club attendant. Since he couldn't swim, he went for the commis chef job under head chef Jeff Bland, "and I never looked back". He progressed to the Champany Inn, Linlithgow, where he was put in charge of desserts at age 20.

Now a father of two, living in Hertfordshire, the biggest part of his job is what he calls conceptualising new desserts and taking them to launch - with the help of 30 in-house developers and seven food manufacturers.

A nod to his Scottish roots is the Cranachan and, just possibly, a Tipsy Laird trifle in the future. Nordic cuisine may also become interpreted for M&S at some point.

But his biggest current influence is Japan, where cheap everyday confectionary consumption is limited but is strong at the high end. It's low in sugar and fat, but extremely beautifully packaged. "The Japanese eat with their eyes, and the absolute precision and detail of their desserts is a joy to behold," he says.

Could it be that the man with a west of Scotland sweet tooth could re-educate our tastes for good?