Britain will not surrender its flag to those who wish to use it as a symbol of violence, fear and division, Sir Keir Starmer has said.

The prime minister said the flag represents “our diverse country” and he will not allow people to feel intimidated on “our streets because of their background or the colour of their skin”.

His comments came after more than 150,000 people took to the streets of central London for a “Unite the Kingdom” march organised by far-right activist Tommy Robinson, and about 5,000 took part in a counter-protest, co-ordinated by Stand Up To Racism.

Earlier, Business Secretary Peter Kyle said marchers were “demonstrating freedom of association and freedom of speech”.

The prime minister said: “People have a right to peaceful protest. It is core to our country’s values.

“But we will not stand for assaults on police officers doing their job or for people feeling intimidated on our streets because of their background or the colour of their skin.

“Britain is nation proudly built on tolerance, diversity and respect. Our flag represents our diverse country and we will never surrender it to those that use it as a symbol of violence, fear and division.”

Police say 26 officers were injured on the day – including four who were seriously hurt – with 24 people arrested.

Speaking about Saturday’s protests on Sunday with Laura Kuenssberg, Kyle said moments like these were “klaxon calls” for those in power to redouble their efforts to address the big concerns people have, including on immigration.

“What worries me most is the divisions in our society and other societies and other democratic societies … it’s not even the left and the right at the moment, ” he told the programme.

“There are figures such as Tommy Robinson that are able to touch into a sense of disquiet and grievance in the community in our society,” he said.

The “small minority” who had committed violence would be held accountable, he told the programme.

Kyle criticised Elon Musk’s comments after he appeared at Saturday’s rally via video link telling protesters to “fight back” or “die”.

“I thought that they were slightly incomprehensible comments that were totally inappropriate”, the minister said.

The tech billionaire also referred to massive uncontrolled migration and called for a change of government in the UK.

Central London saw a huge policing operation for the protests this weekend, with around 1,000 officers deployed.

Assistant Commissioner Matt Twist said officers said: “There is no doubt that many came to exercise their lawful right to protest, but there were many who came intent on violence.”

The protest was largely peaceful on Saturday lunchtime but as the afternoon continued tensions flared.

The Met said some officers had been attacked while trying to keep the two groups apart and dozens of officers were injured, including four who were seriously hurt.

Protesters joining the march, organised by Robinson, became too big to fit into Whitehall, police said, and confrontation happened when officers tried to stop them encircling counter-protesters.

Mounted police officers used batons to push back the crowds and officers were kicked and punched, said the Met.

Three of the 24 people arrested were women and the rest were men, while the youngest and oldest people detained at the protest were aged 19 and 58 respectively, the Met said on Sunday.

Several people were arrested for more than one offence.

The force added officers were working to identify other people involved in disorder with a view to making further arrests “in the coming days and weeks

The Met said on Saturday morning that they were not going to use live facial recognition cameras during the day.

Previously called a “game-changing tool” by Met Commissioner Sir Mark Rowley, the technology is used by the force in “hotspot policing” and major events like the Notting Hill Carnival.