Spacing, in general, has always struck me as problematic in RPGs. The 5' square standard in 3.x is, in my opinion, an absurd figure, albeit one building upon the game's earlier editions. Having several years of hand-to-hand sparring experience, and a fair amount of stage combat and boffing practice, I feel like I know at least a bit about fighting with medieval weapons, and on a practical level, 5' is a humongous amount of space. Looking at how the sizing works out, it seems like combatants are like electrons attached to separate nuclei - they are somewhere within a huge area and occasionally collide at the edge of their respective areas.
Part of the problem is that I don't think a lot of people understand how large a 5' x 5' square is - it's about the size of a queen sized bed. So a typical combat in 3.x features two queen-sized beds next to each other with a single person standing on top armed with a weapon of choice swinging at each other, but not allowed to step onto the other person's bed. Since a lot of us RPGers may not have a lot of experience swinging swords about, instead imagine that everyone has pillows instead of swords and you are embarking in an epic pillow fight on top of these two, side by side, queen beds (a standard pillow is about 2' long, which could be the length of a dagger or short sword, but the way in which one uses a pillow during a friendly pillow squabble gives you about the same reach as you would have with a longsword or katana). Think about the mechanics of a pillow fight (or, better yet, go find someone to fight with for a little while). Chances are, you'll just have one bed (although it could be a slightly larger one, but still far smaller than two adjacent 5' squares), and you'll probably also find that two pillow fighters can have quite the rousing fight on top of that one bed, advancing, retreating, and running all around. On a queen bed, two combatants standing on opposite corners could stand quite comfortably in a combat stance without being able to engage, unless someone moves towards the center. So, two combatants can share a 5' square and fight fairly well, being able to move around and shift positions. In fact, if each fighter were confined to a single queen bed, the space in which both fighters could engage is relatively small.
I'd like to see fighting in RPGs play out like it does in the real-world, with advancing, retreating, side-stepping, and the like. By hand-waving all of that motion away into the massive empty space of the 5' square, combat loses a great deal of the strategic elements that make it appealing, at least to myself. Think of this another way - as a spellcaster in combat, you are evaluating the spacing of opponents across the battlefield combined with an analysis of the terrain to find the place where your spells with have the most impact. In the actual world, a fighter is doing the same thing but with their body - placing themselves in the area where they have the most control of the battlefield. Again, by using a 5' scale, fighters lose the finesse of their movement, since one cannot be halfway in one square and halfway in another.
Now, to make all of this happen, we need (straight-forward) rules for these maneuvers - reasons why a character might advance, retreat, sidestep, or hold their ground. Ways for characters to control the area around them (moving away from the paralyzing 3.x 'Attack of Opportunity').
Our first step, though, is to reduce the size of combat spacing. I use 1 yard hexes in Prodigy. I'm almost 5' 10'', and a 1 yard square is slightly larger in every direction than my typical sparring stance. If I'm unarmed, my attacks have to leave my hex and travel into my foe's, but if I have a weapon, that weapon is already poking into the hex in front of me. Let's talk about the implications of this scale. First, if two fighters each have weapons and are adjacent, their weapons are engaged - they are each within striking distance of each other, so they have to be actively dueling to keep from dying. They have enough room within the hexes to shuffle around a little bit, but any change of positioning requires stepping into an adjacent hex. This brings me to the next reason why I like the 1 yard hex - the average stride is a little less than 3'. So, stepping in any direction places you in a new 1' hex, and a lunge allows you to temporarily step into an adjacent hex to strike someone beyond it. Going back to how much room someone takes up inside the 1 yard hex, my fighting stance takes up almost the entirety of the space, so adding another person would make fighting incredibly difficult for both people sharing the hex. At least in my mind, changing the scale of combat already solves a number of the problems I have with the 5' square.
For games dependent upon the 5' square, just say each 5' square correlates to 1.5 1-yard hexes (or 2 hexes if you want to do slightly less number crunching). If counting, say the first 5' counts for 2 hexes, the second 5' counts for 1, the 3rd for 2 and so on (just like the diagonal rules in 3.x and 4th Edition).
Now, to differentiate advancing, retreating, sidestepping, and holding one's ground. These rules are still in rough draft form as I need to iron out the kinks. Suggestions would be welcome!
Advancing: if you advance on an adjacent character's hex, you are making a driving attack into their area. If they do not repel your assault through skill of arms, they will need to move in order to not be wounded. The easiest movement is a direct retreat in the same direction as the aggressor's advance, but this also allows the aggressor to continue advancing until the defender retreats to a wall or off of a cliff. Alternatively, the defender can step to the side, allowing the attacker past them. Fairly straight-forward.
When the aggressor advances upon the defender, they have a couple of possible responses. They can hold their ground, which is a Fighting Style test made one step harder than normal (in 3.x, the DC would be +5, in 5th Ed, they'd be at disadvantage, etc.) and if they fail they take damage or fall prone. If they retreat from the assault, which is a Fighting Style test made one step easier (DC -5, check made with advantage, etc.) and if they fail, they take damage, are driven backwards a number of hexes equal to the damage taken (damage is usually between 1-3 points), and the attacker may immediately make another advance if they wish. Their last option is to sidestep the attack, which is a standard Fighting Style test, and if they fail they just take damage.
Retreating: Fighting is hard. It is exhausting work. There's a reason boxing matches are divided into rounds with breaks in between. And boxers don't even wear armor! There are also tactical reasons to take a break while fighting - to size up your opponent and look for weaknesses. So to make retreating an important tactical choice, we have to add a combat fatigue system. The basic way this works is each character has a fatigue limit for actual fighting, and going beyond that begins to impair their ability to fight well. I'd probably use poker chips or something for this. Each round your character either attacks or is attacked, take a chip. If the chips you have exceed your character's limit, you start to underperform - maybe a cumulative -1 penalty to everything (so -1 the first round over, -2 the second, etc.).
Looking again at boxing, we see 2 minute rounds for professional women and amateur men and a 3 minute round for professional (male) fighters. Using the shorter time will probably make a better baseline, since we want rules that start with a reasonable expectation for fighters with a minimum of training and extend from there. Now, I claim little experience with boxing apart from hearing other people talk about it, so I went to the interwebs to see some boxing matches. Looking at this
video, we see that they aren't actively engaged every second of every round - they spend a great deal of time maneuvering around the ring. They are only actively trading blows for maybe 3-6 seconds at a time before breaking apart and reevaluating. This maps to my sparring and boffing experience as well.
So, my simple fatigue rules are that a novice fighter can fight for maybe 2 rounds (my rounds are 3 seconds) before getting fatigue penalties. In Prodigy, the Athletics discipline represents a character's general level of physical fitness and can have from 0 to 15 notches (notches are representation of the time and effort spent improving that discipline with quantized skill tiers at 0, 1, 3, 6, 10, and 15 notches), so I'll add one round per notch. A professional boxer likely has 3-6 notches in Athletics, so the could all-out fight for 5-8 rounds or 15 to 24 seconds. Every round spent fighting after that limit incurs the penalty. Now, spending a round out of direct combat allows the character to reduce that fight count by 2.
Sidestepping: This is easy. As part of a standard attack, you have the choice of moving 1 hex in any direction as long as you end adjacent to your opponent. This becomes important when you apply facing rules. I assume that a character in a hex is always facing one of the edges of that hex. That edge and each edge adjacent to it are in the character's full view. Each edge adjacent to these is hard to see, making tests relating to those edges one step harder (+5 DC, etc.), and then the edge directly opposite the direction in which the character is facing is their blind spot and tests relating to that space are two steps harder (+10 DC, etc.).
Standing One's Ground: If you choose not to move, you are focusing more on defense than offense, so defensive actions you take are one step easier (+5 to AC, etc.).