The funny thing about RPGs versus reality is PCs are exactly the kinds of people who would alarm and terrify townsfolk in the real world. They self-professed mercenaries, looters, reavers, thieves, vandals and are generally always looking for a fight. Gangsters. They are actually the last people you’d admit armed through a city gate... or would hire for any decent or heroic purpose whatsoever.

While that is often true, some people actually like to play as noble Tolkien elves or righteous paladins. And when other players want to play as necromancers, assassins and thieves then you have to remind your group that it's a good idea to create characters which won't kill each other in first session or two.

Personally I allow some leeway to PCs, unless I'm hosting a strictly historical setting, because no one in their right mind would allow such people like Elric of Melniboné, Conan the Cimmerian or Fafhrd and Gray Mouser into the city either. In my dark fantasy games it's all right to have sword fights against rich merchant's guards or thieves who haunt the dark alleys (although thieves most likely have just daggers and clubs etc. rather than actual swords). And it's certainly all right to have some weapons with you when you break into temple where priests, with their long daggers, sacrifice people to city's demonic gods.

It depends a lot on the setting and campaign and the style of the
group. In my campaigns such characters would have only rather
short careers and would soon end as a gallows' decoration.

"Yes sir, of course you may stroll around our city in your panoply of very shiny and no doubt very valuable armour. I see you are a man of quality and not no troublemaker. Take care though, some of these streets can be a bit rough, and what with only 0.5% of the city making up the guard... well we can't be everywhere can we? I mean, only last week there was this gent, much like y'self sir, he was strolling along when what should happen? A score of ruffians surrounded 'im, smashed 'is legs in with big hammers and took 'is expensive shiny armour before feeding 'im to the pigs. Well, we was a bit red in the face, but as I say we can't be everywhere can we? Listen, maybe me and may mates could take care it it for you? For a small consideration, of course..."

Well, you sort of answer it yourself, when you state that you need opposition to players tearing up a town: There will be as many "cops" as you as GM consider fitting, within the bounds of reason for the size of a settlement.

Larger towns (5000+) with stored weapons are able to raise their adult, male population (usually one in five in a medieval level of health care setting) to man the city defense if all professional assets are unavailable, but any burg worth bothering about will outfit a city watch, gatekeepers and night watchmen enough for what your story may require.

Not sure if this helps but here are some historical points:

Bear in mind that nearly everyone carried a knife (even children)... it was the primary tool of everyday life, and ultimately pretty useful for eating. That said, the real division is between urban/rural.

In urban areas there tended to be a divide between actual "citizens" and "non-citizens" (who might well have been born in the urban centre and were full-time residents). Citizens were the people with generally most to loose, so they were the ones who tended to join the urban militia and all the armed fraternal societies (and medieval townsmen loved joining clubs!). "Non-citizens" would be "discouraged" from carrying anything too militaristic.

Rurally, although occasionally the bearing of arms (beyond the usual knife or necessary tool) by someone without their lord’s explicit permission was punished by outlawdom, such restrictions were rare. People tended to know everyone about in their community and acted accordingly.

More typically "arms" were banned at certain locales or whilst attending specific events. An example being Louis VI of France’s prohibition on the bearing of arms in markets or towns, and 14th and 15th century Bavarian Landfrieden legislation that forbade subjects from carrying weapons on ceremonial occasions.

The chief reason for this being that weapons were useful, and although many had them, only soldiers and the rare manorial watchmen generally carried them. Indeed ownership of weapons was accepted, the need to raise militia units making this a necessity. Honour-feuding by populations (more common in rural populations without easy access to senior courts) also often meant arms were carried, this being seen as an extra-legal method of pursuing justice, and although frowned upon was not often stopped.

Knife-fights were pretty common, although in real-life they often don't last very much longer than initial blows... which is still typically the case today. Also drink was drunk by nearly everyone, but social and religious pressure also acted to keep tensions "sort-of" under control... generally.

It was the big festivals and fairs where people got excited and drank more than usual, that were the real problems (hence the rules about carrying arms at them).

_________________
The desire for a "definitive, ultimate answer" is, in fact, classified by modern psychiatric medicine as a mental illness.