AS resignation letters go, it was pretty devastating.

Delivered at just after 5.30am, Oliver Dowden, now the former Conservative Chairman, told Boris Johnson the UK Government could “not carry on with business as usual” and “someone must take responsibility”.

And Dowden, saying he shared the “distress and disappointment” of Tory voters by recent events, ended his brief letter of resignation by stressing how he would “as always, remain loyal to the Conservative Party”. Not, that is, loyal to the PM.

The letter was worded very carefully and open to interpretation, which most people will read in one way; Johnson should go. In the hours after it dropped, Dowden was keeping conspicuously silent.

The PM’s initial reaction 4,000 miles away in Rwanda was expected; he would “listen” to voters but would “keep going”. His deputy, Dominic Raab, dismissed the bad results as being, in part, due to “distractions” like Partygate. So, the PM’s behaviour is now regarded as a distraction.

A party source revealed how Johnson was swimming in his hotel pool at 6am Kigali-time and then received the Dowden thunderbolt before a 7am meeting. He phoned Rishi Sunak, the Chancellor, and Chris Heaton-Harris, the Chief Whip.

The source echoed Raab and attributed some of the blame for the electoral losses on the “nonsense” of the media’s “endless reportage and Kremlinology of partygate”.

The opposition parties’ reaction to the two crushing by-election defeats - the one at Tiverton and Honiton saw the Tories lose by a record 24,000 votes - was predictable.

After Labour retook Wakefield with a majority of 4,925 on a swing of 12.7% from the Conservatives, Keir Starmer said: “Wakefield has shown the country has lost confidence in the Tories. This result is a clear judgement on a Conservative Party that has run out of energy and ideas. Britain deserves better.”

For the Lib Dems, which have now taken three safe Tory seats in this parliament, their leader Ed Davey said: “This should be a wake-up call for all those Conservative MPs propping up Boris Johnson. They cannot afford to ignore this result.”

Read more: Nicola Sturgeon: By-election defeats 'massive, humiliating vote of no confidence' in Boris Johnson

Indeed, only 40 Conservative MPs have a bigger majority than the one the party had in Tiverton and Honiton. There is nothing that concentrates the mind of backbenchers and frontbenchers more than their own survival.

Lord Barwell, Theresa May’s former Chief of Staff, described the Devon result as “catastrophic” for the Tories, telling Sky News: “It’s one of the safest Conservative seats in the country. It’s a strongly Leave-supporting constituency.

“So, for the Liberal Democrats to be winning there, and winning comfortably, it means that there’s a whole swathe of seats across the south of the country that are vulnerable.”

Senior Tory, Geoffrey Clifton-Brown, made clear the backbench 1922 committee, of which he is Treasurer, would have to go away and consider some “difficult decisions” – which sounds ominous for Johnson.

Under current party rules, the PM, having won the recent confidence vote, cannot be challenged for another year. The difficult decision now for the 1922 is whether or not it changes the rules to allow another vote.

Even some Johnson critics are deeply uncomfortable about this. But it should be noted that in the confidence vote an estimated three-quarters of Conservative backbenchers voted to get rid of the PM and the 1922 is, supposedly, a backbench committee.

But the key thing now is not so much what backbench Tory MPs think but what the Cabinet thinks and, more vitally, what it does.

Thus far, through all of the PM’s travails, its members have steadfastly stayed loyal. But while two by-elections were factored in, the scale of the Tiverton and Honiton defeat will have seriously shaken Johnson’s colleagues. Telephones will be buzzing across Whitehall and Westminster this morning.

They know that the two votes were, in part, a referendum on Johnson’s performance; a repeat anything like that the one in Devon would devastate the party in 2024.

Of course, all mid-term by-election results do carry something of a health warning as often a defeat for the governing party facing serial challenges is reversed at the following general election.

However, the scale of the Conservative defeat, especially in Tiverton and Honiton, will cause nerves to seriously jangle throughout Tory ranks.

One detail in particular that will worry Conservative MPs is that the figures show that Labour voters in Devon tactically switched to vote Lib Dem and sealed the Tory candidate’s defeat. If this were replicated elsewhere in a general poll, then many Conservative candidates would lose in 2024.

Read more: Boris Johnson suffers double by-election humiliation

However, Tory HQ may take a crumb of comfort from the scale of Labour’s victory in Wakefield, which was not overwhelming and may leave Tories with some hope that the electorate is not yet enamoured with the alternative and that, in the two-year run-in to the 2024 election, things can be pulled back.

There is one view at Westminster that given how damaged Johnson is and given how things are economically and politically troublesome, that the party would be better to keep him in situ for now, so he can take all the flak, and then to get rid of him when there is light on the horizon to help his successor have a fighting chance of winning the next election.

But, amid the summer of discontent and the cost-of-living crisis, the political landscape is already shifting away from the Tories and they may not have the luxury of waiting.

Back in 1990 when things turned against Margaret Thatcher, she was famously abroad when the metaphorical knives were unsheathed. As Johnson is due to be away for another seven days, Cabinet colleagues will be privately contacting each other about the next step and whether it’s time to unsheath them once again.