THE nearest most of us come to horror or terror is in nightmares. However that may not be as comforting as it sounds. In actual situations of extreme threat and danger people often react with surprising calm and self-possession, as if our systems are wired to close down imagination and focus on doing something useful. But in nightmares we are exposed to the experience of horror in its essence, and it comes at us full force – out of the dark.

Of course, people’s nightmares are different, tailor-made as it were to our own sensibilities. For some, it is the cold terror of pursuit by a remorseless force; for some it is confinement or entrapment; for some it is a haunting of malign, unspeakable evil. Recent events show that clowns can be the embodiment of malignant fear. Often, the cold sweats are produced by a more low-level distortion of everyday life – the unprepared-for exam, the disastrous interview, humiliation on social media, or the sudden death of a loved one leaving our life in ruins. Sometimes, we are lost in these dreams, unable to grasp what is happening or find a way through – the kind of inexplicable loss of control that incipient dementia must bring to sufferers.

I was reminded how fertile our dark inventions can be when reading the nearly 300 entries for this year’s Scottish Storytelling Festival’s ghost story competition. This was inspired by the similar contest 250 years ago, on the shore of Lake Geneva, when in the grip of stormy weather Mary Shelley trumped Byron and her poet husband by birthing the tale of Frankenstein. That story is one of the few modern inventions that achieved mythic status, reflected in countless adaptations and retellings.

I am not sure that this new competition has unearthed another Frankenstein, but it certainly reveals a wide range of takes on ghostly terror. There are horrors aplenty reflecting all the tropes of pursuit, confinement and extreme violence. There are also many clever variations of nastiness, including new ways in which digital technology allows us to be intimately vindictive at a safe distance – as if being able to be "a ghost in the machine" brings out a previously unexplored darkness in human nature.

Yet, granting all that, many of the ghost tales are about hauntings which are not malign or demonic. They are stories in which some kind of spiritual connection persists between the living and the dead. This may be expressed through a place or an object, and it is often consoling or informing, allowing us to understand something that was previously mysterious and troubling. In this way the ghost tale goes into a different dimension that is not dark but illuminating.

Also there is a surprising amount of humour in ghost stories. The narrators set us up for the dark shadows but then pull the rug from under us to comic effect. Traditional storytellers, such as the Scottish Traveller Stanley Robertson, were particularly fond of this device. In live storytelling, a look or tone of voice can achieve subtler yet more immediate effects than words on a page.

Traditional stories from the world over associate the power of dream or fantasy with both dark and light. A typical South American tale from the Orinoca river delta describes how a chief’s daughter woos "the keeper of the light box" in order to bring light to her people. At that time everyone lived in darkness, depending only on firelight. The chief's daughter is successful, bringing back the "Itiriti", the rush woven light box. However possession of this gift brings endless and innumerable visitors to the Warao people. They want both the light by day and the dreams that the box confers by night. So her father takes the light box and hurls it into the sky, creating both the sun and the moon, through which everyone can experience daylight, and the magical half-light of night dreaming.

Because of the international reach of the Scottish Government’s Festival Expo programme, many South American storytellers are coming to this year’s Scottish Storytelling Festival to work with Scottish storytellers, so reflecting a continent rich in imaginative dreaming. They reveal that the power of dreams is two-edged, bringing a capacity to imagine and invent, but also dark dread. Yet can we have one without the other? And is that why many desire to go into the darkness?

That urge is especially strong in science fiction. "Into the dark" features in innumerable film and book titles, because the journey or adventure requires an extreme challenge. That involves going out into uncharted immensities of space where only the distant light of stars offers the allure of vision, dream, discovery and renewal. Meanwhile the dark is full of phantom emanations of dread or danger, which must be faced and defeated if the quest is to be fulfilled.

Of course, this is a variation of the hero or "shero’s" journey beloved of the traditional storytellers. The journey in their case may be down into the black depths of earth, or into the dark recesses of the forest, or out into the sky world. But the mission is the same – to meet some challenge or enemy or mystery, and to return with a vital gift or discovery. But that enlightenment cannot be gained without the descent into darkness. The two belong together, as in the power of dreams.

All of this may be a profound insight about how we are as humans. "Without the dark no light" is as true for our psychological make-up as it is in the physical world. We cannot live solely on the light surface. That leads us into a bubble world, disconnected from our own natures. We need the energy and fertility of the dark within – what has been called by some the unconscious, though in truth there is a constant traffic between conscious awareness and what lies beneath the surface. It is that two-way movement between light and dark that the hero’s journey re-enacts in whatever genre.

The same journey through dark to light is embedded in the natural world, because of seasonal change both in the northern and southern hemispheres. Hallowe'en began as a seasonal festival marking the time when darkness begins to reign over a winter world. In the old Celtic calendar, that condition prevails through the hidden promise of the winter solstice, the first signs of spring nurtured by Bride in February, finally reaching the celebrations of Beltane or Mayday when light is restored as the ruling influence. Through these festivals, the natural world becomes a theatre of cosmic cycles within which we live out our own humanity.

Hallowe'en therefore is much more than "trick or treat", but it does involve fear. Our ancestors saw the time of transition between light and dark as dangerous. If the correct ceremonies were not performed then perhaps the delicate balance of life would shift and uncontrollable darkness invade our world to destroy us. The veil between the natural and the spirit worlds becomes paper-thin, but it is a heroic balancing act that keeps us on the pathway through. Guisers, for example, performing the ancient Mummers' plays, went masked so that their identity would be concealed from any hostile ghosts.

Later when political and religious authorities wished to suppress the older communal ways, Hallowe'en was maligned as an expression of malevolent witchcraft and "treating with the devil". There may be more to "trick or treat" than meets the eye or ear. It was in America especially that the Puritan founding fathers demonised witches and used religion as a means of social control. They felt themselves to be on a dangerous frontier requiring religious discipline and moral order. Interestingly the beliefs and customs of the Native Americans they encountered bore more than a passing resemblance to the old Celtic ways they had suppressed at home. Religion became a justification for conquest and extirpation.

The older understanding of Hallowe'en as Samhuinn, the Celtic New Year, still has something to give the 21st century. That is because it provides a collective expression of that balance between dark and light. This meets our psychological needs but also our connection with the natural world. The idea that "man" – note the gender – could master the universe and bend it to his will has proved a destructive illusion. Re-aligning with the ways of Mother Earth now appears not only attractive but sane.

Going into the dark together is not threatening but a wise necessity that prevents us living superficially in the artificial bubble of consumerism. But the critical point is the "together". We are not made to face all the challenges of human existence on our own. The most chilling experience which the terror school of ghost tales evokes is being left alone to face whatever horror looms, without help, companionship or human contact. Trapped, entombed, cornered, unaided. The old wisdom is that we should face inner and outer dangers together and journey through dark to light.

The problem we face in the developed world is that, despite our unprecedented material wealth, and our multiple communication technologies, more people than ever feel isolated. Worst of all, as several of the ghost tale competition entries show, the digital media are feared because they are used as instruments of threat or exposure. So tragically we turn our own inventions against ourselves. How could Hallowe'en possibly help?

Who were the Hallowe'en spirits or ghosts who crowded around the living in the old Celtic New Year? They were sometimes thought of as "the ancestors" and sometimes as "the dead". Though the passage from dark to light was dangerous – because the ancestors might prove hostile or threatening – it could also be strengthening, especially if the ceremonies went well. The purpose of the rituals, stories, dances and songs was to sustain a bridge between living and dead. The ancestors were not conceived of as lost or past, but as continuing in some parallel plane to which the living could remain connected. In this way the accumulated experience and energies of many generations could be tapped to support the future.

This seems a long distance from our contemporary way of death. Instead of a collective experience, death has become a specialised kind of medical event dealt with in private. This leaves the bereaved to cope with the whole process in isolation. Such an attitude was inconceivable in traditional cultures. There, people gathered round the dying to ease their passage and support the bereaved, in order to ensure continuing bonds between the living and the dead.

It is no accident that Hallowe'en is followed in the traditional calendar by the Christian festivals of All Saints on November 1, and All Souls on November 2. These affirm successively the ancestors, in Christian terms, and the dead. These Christian ceremonies continue one of the core purposes of Hallowe'en – keeping the dead alive in our memories, emotions and spirits. It is fascinating that despite the desire of religious institutions, past and present, to exercise social control, at a deeper level Christianity continued some of the core features of primal religion. George Mackay Brown, a poet of seasonal ceremony, said that Christianity at its best brought new light to give ancient patterns fresh depth and self-awareness.

In a more secular age it is heartening to see initiatives such as the Absent Friends Festival, which encourages new expressions of remembrance. One of their ideas is an Absent Friends Supper, a celebration of our loved ones with food, stories, perhaps music and song. It is a little like a Burns Supper, though with a focus on those we have lost, but whom we wish to "keep alive" together as friends and family. This year’s Storytelling Festival remembers and celebrates Dario Fo, who died recently in his 90th year. He became a friend to Scotland, and gifted his artwork to the Festival illustrating the power of dream.

Without such deep connection our humanity easily loses its roots, prey both to lonely alienation, or to manipulated false emotions. We need to keep our older traditions alive, enacting and interpreting them in new ways. This year the Storytelling Festival once again brings new insights to bear on some very old expressions of fear, but also of wisdom and love.

Donald Smith is director of Tracs, Traditional Arts and Culture Scotland and of the Scottish International Storytelling Festival, which runs until October 31, under the theme, Festival Of Dreams. Six winning tales from the ghost story competition will be told at the National Library of Scotland on Monday 31st October 2016, All Hallows’ Eve, as part of the event