Jamie Byng rolls into Edinburgh's Canongate office dragging a case on wheels behind him, and though he still has the tousled, laid-back air of someone who might have just come off a big party night, that’s not where he’s at. He spent the morning on an early train from London and hasn’t drunk alcohol for months. For 21 years he has been coming into this publishing company as managing director, so he’s a little older now. Perhaps, even, he has come of age.

Canongate itself, set up in in 1973, is of course exactly twice 21, and Byng himself, at 45, a little more than double. If he seems forever youthful it’s because he is still fairly young. Of course he has aged a bit – there is the strip of grey in his rugged beard, the odd line across the forehead – but Byng still has the overall air of being one of the young dudes. When he has his photograph taken, he still stands, smoking and squinting into the sunshine as if he might be channelling Bob Dylan. Does he feel very different from that 25-year-old who audaciously instigated the buy-out of Canongate after just two years working there? “Not that much,” he says. “I feel different in ways, but you grow up a little and see things in a slightly different way. Becoming a dad again to two younger children, for instance, is just quite an extraordinary experience."

One of the striking things about Byng is that he did all the big things young – and not just the buying out of Canongate in 1994. He also had a family – two children, Marley and Leo, with his first wife, Whitney McVeigh – while still in his twenties. Now he has another family, Nathaniel and Ivy, who are six and three years old, with second wife, literary agent Elizabeth Sheinkman. There’s an audacity to all that. "I feel glad,” he says, “when I look back in hindsight that I did certain things when I did, not least becoming a dad. That was very important, and remains so. Four times and with two amazing mothers.”

We talk about the fact he has been at Canongate for 21 years now, a subject he will discuss at an event at the Wigtown Book Festival on September 25. Really, though, he says that coming-of-age figure means nothing to him. “I don’t really think about time like that,” he insists. “Every year that goes by is another one in which you hope you’ve done a whole lot of things that you can look back on and think were worth the time spent on them.” Nevertheless, he notes, “its’ been a rollercoaster ride in so many ways."

Still, when prompted, he does look back, and what he picks out were the beginnings of big relationships. The moment in 19997 he met Michel Faber, whose many books, including this year’s The Book Of Strange New Things, he has published. “I couldn’t have known that we would end up publishing for as many years together,” he says. There's the time he had lunch with a woman in New York who started pitching Life Of Pi, the book that would make Canongate’s fortune. Among the books he is particularly proud of in the last year is Matt Haig’s Reasons To Stay Alive, a memoir of depression which was 15 weeks in the bestseller list. This too is a relationship, he says, that has been cultivated over time.

Canongate still seem to be riding the crest of a wave, and publish Martel’s latest novel, The High Mountains Of Portugal, next year. “There are more opportunities now than ever for us as a publishing house,” Byng says, “and for me as a person.” On some levels, work life and the personal seem barely separate for him – at one point, in London, he lived above the Canongate office. “I don’t see any clear divide between my personal and my professional life. Nor would I say there is any clear divide between work and downtime. It’s all some sort of continuum till you die. And all you hope is that you don’t die in some protracted and unpleasant way.”

His health however is currently good – though there are some habits he has changed. For the past few months he hasn’t touched alcohol, an experience, he describes as “really nice”. This, bear in mind, is the man who in a 2005 interview said, “I regard myself as quite bulletproof in some ways. I can get absolutely f***ed and then get up and work." Now he says, "Yeah, I’m still smoking and drinking quite a lot of coffee but I’m relatively clean living.”

Byng no longer owns a flat in Edinburgh. While here he stays with the filmmaker Mark Cousins. On one of his birthdays, he recalls, Cousins gave him a birthday card. “On it was printed this beautiful line by French filmmaker and philosopher Robert Bresson: 'Make visible what, without you, might perhaps never have been seen.'” For him, that seemed very appropriate, this “making visible”. “You are literally,” he says, “putting into ink the ideas that someone has and that might never have been seen.”

Canongate's London and Edinburgh offices are very connected. Watching over us, at the end of the meeting room table, is a large video-conferencing screen – and it almost feels as if it’s there, London, through the portal. How Scottish is Canongate still? They have an office in London and Byng mostly works there, but still the bulk of the staff work out of Edinburgh. Byng says it is critical to him that Canongate remains “principally based in Edinburgh”. “It just feels it’s one of the things that is part of our DNA – and it’s not that we publish books primarily for an audience that’s based in Scotland or, obviously, exclusively Scottish authors. But it’s part of who we are.” That “exclusively” is, of course, an understatement. New books by Scots on Canongate lists have been few in recent years: virtually the only publications of note being Alasdair Gray’s Independence and Anne Donovan’s Gone Are The Leaves.

Byng has long had an outsider air about him. It comes out frequently in our conversation, his siding with the non-establishment. He likes Jeremy Corbyn, relates to the disillusionment of the younger generation, moans about Westminster and complains about the brazen way those in power abuse it. I can’t help thinking Canongate’s rootedness in Edinburgh must chime with that outsider side of his personality. “Yeah,” he admits. “It is also that cliché: if it ain’t broke don’t fix it. It works well for us being here.”

Byng wangled his way into a work experience job at Canongate in 1992, by sending in a letter to Stephanie Wolfe Murray (co-founder with her husband Angus of the publishing house) with a packet of Mini-Munchies and a flyer for his reggae and rare-groove club night, Chocolate City, attached. In the first two years he worked there, he recalls he “got the bug of publishing completely”. “It was an amazing first entry point, not least because of Stephanie who was and still is an inspiring and amazing women.” He and Wolfe Murray remain in regular contact. “I’m incredibly fond of her and enormously grateful because she was the one who first kind of took me on here.”

But where will the publishing industry, let alone Canongate, be in another 21 years time? For all the doom and gloom that often seems to lurk around the future of publishing in a digital age, Byng believes the book has a future. “The physical book too,” he says. “And actually that is already being borne out by the statistics we have already seen. The e-book that grew massively in the years 2010 to 2013 really plateaued in 2014 and is continuing that way. That’s reassuring.”

Nevertheless, he also says that he no longer even thinks of himself as a publisher “in the traditional sense of the word”. “We are working in all sorts of areas in a more strategic way, going to areas that we weren’t ten years ago.” Last year they created a separate business called Letters Live, “a celebration of the enduring power of literary correspondence” which came out of Canongate Letters Of Note and More Letters Of Note books. It is event-based, revolving around live performances of readings of the letters by actors, musicians and writers, including Benedict Cumberbatch, Gillian Anderson and Matt Berry, and watchable online. Projects like these, Byng says, are "just ways that you are directly connecting with the audience. It’s really about putting stuff out there which we think will make people think about who they are and who other people are and what it means to be human.”

Though he confesses, he is not “a political animal”, he does describe himself as “definitely an activist”. Recently he helped pull together the signatories, all cultural figures, for a letter published in the Guardian calling for the UK government to wake up to the urgency of the refugee crisis. Last Saturday, he recalls, he took his three year-old daughter Ivy on her first ever march and rally for refugees. “But,” he says, “I think there are lots of ways one can be an activist. To me, that’s what we were doing when we published Mohamedou Ould Slahi’s Guantanamo Diary earlier this year. That is a book that is absolutely challenging something that continues to go on, which is just immoral and illegal on so many levels, and the fact that Guantanamo is still imprisoning all these people, none of whom have ever been tried for anything.”

So, how did the publisher of Alasdair Gray’s Independence feel about the referendum? “I was very sympathetic to the Yes argument. And continue to be so. Generally speaking I was supportive of an idea that less control of what goes on in this country was being determined out of Westminster. That feels wrong – particularly with the people who are in Westminster at the moment.”

Byng has often been caricatured as a child of privilege, educated at Winchester School, described once, in the Guardian, as a “posh huckster”. Privilege is something we as a society talk a lot about these days, and it still dogs him a little. “All you can do is, if you are privileged, f***ing do something with that,” he says. He has, he adds, an uncomfortable relationship with the very idea of privilege. “Of course, I was incredibly privileged in terms of the education I had and the fact that I had a very settled childhood and grew up in a very loving family.” But he notes, there are type of privilege and “lots of people who go to private school have miserable experiences. And lots of people have really great experiences going through the state system.” His two eldest children, Marley and Leo, both go to a state secondary, Queen’s Park. “Marley’s just done her A Levels there – that’s a great state school. And I think she’s had a fantastically privileged education but it’s not a private one.”

He remains a “massive reader” himself. “I think I’m reading as widely now as probably I’ve read at any time since I was an undergraduate.” He also reads frequently to and with Nathaniel and Ivy. “It’s a beautiful thing to watch,” he says of Nathaniel, “when you see your child taking those first steps and starting to flap their wings as a reader, and suddenly they’re kind of airborne and they can read chapter books on their own.” His son, he says, however, has a “pretty high quality threshold... If he doesn’t like something he'll throw it aside and I’m glad to say he’s got pretty good taste in books, like he’s already got incredible taste in music, and he’s only six.” Byng, ever the music lover, is also “homeschooling” his son “on the music front”. “Because you’re never going to get the kind of music education that I can give him – so already he’s totally into David Bowie, totally into The Beatles, totally into Bob Marley."

One thing that’s clear is that Byng, on this 21st coming-of-age, has no desire to leave the industry or Canongate itself. I suggest that perhaps his life in publishing seems like a relationship that perhaps he never sees ending. “You’re right to observe that. That’s the way I see it. I certainly feel that at this point in my life I can’t imagine not wanting to be publishing books or working with writers not being a central part of what I spend my time doing. I hope I’ll always do that through Canongate.”

Being independent, remains key to him. “I love the fact that we’re based up here and we publish the people that we do. We are, I think, a formidable small but really dynamic team who, in certain respects, can do things better than any other publisher around.” Hence, he has no desire, to work at “some big publishing house.”

“Why would I want to do that when we’ve created what we have here and we are totally different from any other publishing house? We have a kind of identity that’s distinct from anything else. Independence seems to be hard-wired into my system. It’s deep in my blood now.” Here he starts throwing out metaphors. “It’s like I don’t want to be suddenly domesticated. I don’t want to be in the zoo.”

Jamie Byng: 21 Years At Canongate is part of the Wigtown Book Festival on September 25. For the full programme, see www.wigtownbookfestival.com