BENJAMIN Franklin was much impressed by Richard Oswald.

According to the American founding father, the son of a Caithness Kirk minister had about him of "an air of great simplicity and honesty". The likes of James Boswell and Laurence Sterne also enjoyed the merchant's company.

To his contemporaries he was, as the author Adam Hochschild has written, "a wise, thoughtful man who embodied the Scottish virtues of frugality, sobriety, and hard work". Oswald was a scholar of theology, philosophy and history. He collected art, particularly Rubens and Rembrandt, and gave handsomely to charity.

Oswald, who learned his trade in Glasgow, also represented Britain in negotiations with the Americans after their war of liberation. He was the cosmopolitan epitome of Enlightenment success. But when he wasn't busy with good works, Oswald waded in blood.

The precise number of deaths that can be laid at his door is impossible to calculate. As the leading figure in Grant, Oswald & Co, he had investments in each corner of the "triangular trade". In his own name, Oswald trafficked at least 13,000 Africans, although he never set foot on their continent. By the time he bought Auchincruive House and 100,000 acres in Ayrshire in 1764, he was worth £500,000. Writing in 2005, Hochschild thought this was "roughly equivalent" to $68 million (about £44m). This is conservative.

Oswald was remarkable, but not unique. Where Glasgow and its merchants in sugar, tobacco and human life are concerned, there are plenty of names and no shortage of monuments: Dennistoun, Campbell, Glassford, Cochrane, Buchanan, Hamilton, Bogle, Ewing, Donald, Speirs, Dunlop. One way to understand what they wrought is simple: take pleasure in the city's architecture today and you are likely to be admiring the fruits of slavery.

Glasgow is not alone in that. London, Liverpool and Bristol also have their stories to tell. Edinburgh's once-great banks grew from foundations built on bones. The first Scottish venture into slavery set out from the capital in 1695. Montrose, Dumfries, Greenock and Port Glasgow each tried their hands. In the language of the present age, they were all in it together.

When commerce was coursing around the triangle, most of polite Scotland was implicated. The nobility (and country) rendered bankrupt in 1700 in the aftermath of the Darien Venture was by the mid-1760s contemplating big elegant townhouses and 100,000-acre estates. You could call that a reversal of fortune. Contrary to self-serving myth, it did not happen because of "frugality, sobriety, and hard work".

Certain things need to be remembered about Scotland and slavery. One is that the mercantile class got stinking rich twice over: despite fortunes made from stolen lives, they were quick to demand compensation when slavery was ended in 1833. Britain's government decided that £20m, a staggering sum, could be raised. In his 2010 book, The Price Of Emancipation, Nicholas Draper reckons Glasgow's mob got £400,000 – in modern terms, hundreds of millions.

Compensation cases also demonstrated that Scots were not merely following an English lead. According to Draper, a country with 10% of the British population accounted for at least 15% of absentee slavers. By another estimate, 30% of Jamaican plantations were run by Scots. For all the pride taken in the abolitionist societies of Glasgow and Edinburgh, the slave-holders did not suffer because of abolition. They were "compensated".

And that wasn't the worst of it. Thanks to Hollywood movies, the slave economy of the American South is still taken as barbarism's benchmark. Few realise that the behaviour of Scots busy getting rich in the slave-holders' empire was actually worse – routinely worse – than the worst of the cottonocracy. You need only count the corpses.

By the time slavery was brought to an end in America, the country's 400,000 trafficked people had grown to a population of four million. In the British West Indies, only 670,000 survived from two million imported souls. In the American South, slaves were valuable and bred. We worked them to death then simply imported more to keep the sugar and thus the money flowing.

Liverpool, London and even the Church of England have apologised for the years of slave trading. Unlike centuries of grief and murder, an apology costs nothing. So what does Scotland have to say?

Last week, the writer Chris Dolan and theologian Dr Robert Beckford could be heard saying that the time has come for our apology. Beckford, of Canterbury Christ Church University, reportedly suggested that the impending Glasgow Commonwealth Games would be a good moment. All I add is that a statement of regret and remorse should come, as history demands, from the entire country.

You can hear the objections. Why should anyone in the 21st century be held responsible for the deeds of others – deeds we would not sanction and cannot retrospectively prevent? What is this "culture of apology" that demands confessions of guilt from the innocent and too often creates excuses for the modern leaders of failing countries in the developing world? What's the point?

A few answers. An apology costs nothing beyond honesty, and honesty is never a bad thing. A country that does not take responsibility for its past cannot prosper in the present. History is neither optional nor dead – recognise the truth of past horrors and you remind yourself that horrors continue, with our complicity.

Do you know how the things you eat were grown and picked? Have you worked out why so many of the consumer goods that keep the wheels of global commerce spinning are so cheap? How did this smartphone or that T-shirt get to you? Are you stupid enough to think slavery no longer exists? A statistic to be going on with: the International Labour Organisation calculates that 126 million children around the world labour with no choice in the matter.

Scotland being Scotland, there is a poetic twist in this tale. Not all the slaves in the 18th century were black. In fact, in Barbados at one point 21,700 of 25,000 held were white. A great many of them, then and afterwards, were Scots who happened to be poor, homeless or political nuisances. Rounding them up and selling them off, especially after the '45, was routine. In the language of the time, and for obvious reasons, these were "redlegs".

A modern, multicultural Scotland would have to apologise to itself several times over, to its white people as to its black and brown, to begin to set history straight. A country that refused the gesture would be a comfort to the shade of Richard Oswald and all those who carry on his bloody work.