Pull your chair up closer to the fire, and let me tell you what makes a good ghost story. It’s that season again – for folk like me, the most magical, dreamlike time of year. Christmas is joyous, yes – a festival of family and fun. But Halloween marks a truly different pace – set aside, like the chaotic days of a renaissance carnival, for turning the world upside down, looking deep within ourselves through the wonder of storytelling, and finding out what really makes us human via the simple mechanism of scaring ourselves silly.

I’m ghoulish by nature. I’ve been devouring horror stories my entire life. One of my novels was a horror yarn, and I’ve written macabre little ghost stories for children and adults. This last month I’ve been re-reading the best spook stories of the last few hundred years – at least one a day – and listening to classic horror tales on my earphones during long autumnal walks in the woods. My dreams have been haunted – and I’ve learned a thing or two about what makes a ghost story work.

First off, we need a good teller; the best tales are written as if the author is sitting beside you, fire blazing, drink in hand, with the wind rattling the windows. EF Benson – now best remembered for his Mapp and Lucia novels rather than his sparkly dark ghost fiction – makes you feel as if you’re right there with him as he talks to you. In his creepy little yarn, The Room in the Tower, you can almost hear his voice tremble. That same story contains another important ingredient: an hypnotic style of writing. The best ghost fiction both lulls and terrifies – it has a ballad quality. Just listen out for the refrain: "Jack will show you your room, I have given you the room in the tower.”

Benson – a genius of the form, though he is – falls somewhat short in one respect in this tale. Every chiller – whether on film or in literature – needs a damn good, neat ending. Too often, horror writers find themselves with an incredible central concept, but no way to resolve the story satisfactorily. Benson here ends his tale a tad abruptly.

MR James, on the other hand – who held Benson rapt at university with his Christmas ghost readings for friends and students – spun faultless endings. Casting the Runes has a great central premise (a demon summoned from beyond to pursue you to death), is written with quiet grace, but unfolds to a relentless and brilliant close.

The Herald:

A charming little ghost compendium has just been published called The Wrong Turning. The editor Stephen Johnson contends that many of the best chillers are based on the protagonist making a faulty decision – a metaphoric “wrong turning”. He’s quite on the money here. Take The Monkey’s Paw by WW Jacobs – one of the most influential ghost stories ever written. Everything swings on the two central characters, a bereaved mother and father, taking a mortally foolish step while grieving for their lost son.

One of the most sorely overlooked of all horror writers, Robert Aickman, built a literal wrong turning into his queasy masterpiece, The Hospice, where a middle-class motorist goes left instead of right and finds himself in the kind of hotel nobody wants to visit. The Hospice, in my view at least, is undoubtedly the greatest 20th-century horror story. It’s a drug dream that will stay with you until the day you die – and you’ll never quite work out what it means.

That sense of confusion – of something half-glimpsed in shadow from the corner of the eye – is crucial in all good ghost literature. You want a taste of what psychologists call the “uncanny valley” to really get your heart racing – the idea that something isn’t quite right but we can’t put our finger on it. Scientists say a good current example of the “uncanny valley” is that sense of creepiness we get when we see a modern robot covered in latex flesh with convincing human features. It looks like us, but we know it’s not us – and that “valley” between perception and understanding is where the terror lies.

HP Lovecraft remains the king of the uncanny through the tales that comprise his action-packed Cthulhu Mythos. Why is Wilbur Whateley quite so goatish-looking in The Dunwich Horror?

The Herald:

In the anonymous A Tale for Twilight – what is the nurse really seeing at the foot of her patient’s bed? That yarn provides the title for a new collection, Tales for Twilight: Two Hundred Years of Scottish Ghost Stories. Contained within its pages is a neat little yarn – Ronald Duncan’s Consanguinity – which explores the idea that lies at the intellectual heart of all good ghost literature: our obsession with, and attempt to, unravel the past. What is a ghost after all but the past come back to haunt us due to matters unresolved in life?

Consanguinity, again like all the best ghost fiction, picks at the darkest side of humanity – in this case the underlying theme of incest. The ghost story is often a bait and switch: look, it says, here’s some spooky distraction for you – only to pull the rug from under you and force you to contemplate some horrific, twisted truth.

Unsurprisingly, then, women often make the best writers of ghost stories – a fact too little acknowledged. The Yellow Wallpaper by Charlotte Perkins Gilman, Black Dog by Penelope Lively (both in The Wrong Turning anthology) and The Girl I Left Behind Me by Muriel Spark (in Tales for Twilight) all present devastating takes on the oppression of women which masquerade as spooky little diversions of gothic fiction.

In truth, the secret to what makes a great ghost story is as hard to articulate as the secret to what makes a good joke. It’s a gossamer, intangible craft – a gift which makes artists of a few, and leaves the rest of us as their audience. But there’s one surety: ghost stories must remind you that you’re human and one day you’ll die. Ghost stories are momento mori laid down on paper.

If you doubt me, pick up a copy of EF Benson’s The Bus Conductor – “Just room for one inside, sir!” – and tell me you didn’t contemplate the inevitability of your own fate as the story closed.

The Wrong Turning: Encounters with Ghosts edited by Stephen Johnson is published by Notting Hill Editions, £14.99; Tales for Twilight: Two Hundred Years of Scottish Ghost Stories selected by Alistair Kerr is published by Polygon, £10