Game Networking Improvement

[etigah]

Hi,

I have brought up the issue of handling lag before and suggested two ways to improve it. I am again bringing the issue up for more discussion, and suggesting a new solution.

It is a three component solution, two of which are known since I made them before, and even created POC implementations for them. The three part solution briefly is :

1- Add visual indication of the players geographic location , which is displayed on radar. For simplicity there will be three main regions, US, EU, and AU. The indicator can be something unobtrusive such as a small horizontal colored line above the tank radar object.

2- Add a form of lag compensation. What I suggest, and have a POC implementation for, is to alter bullet speed slightly based on players combined lag. For instance, if combined lag is 400ms, increase bullet speed so that it would take 150 ms less to reach end of life for a shot. This can be tuned so that it has minimal effect on game play. Combined with the first suggestion, players will easily adjust to bullet speeds based on the geo location indicator.

3- This is a new suggestion, and I do not really know if it is feasible. Hosting a game on multiple servers simultaneously. That is, each player connects to multiple servers, say a US and a EU server for instance. Each event the client sends to both servers, and both servers send to all other clients. The events have unique ids, so whichever server manages to deliver the event to another client faster, the client then discards the other server's event with the same id. The obvious benefit is that players in the same region will experience very low combined lag always. and overall average combined lag will be less. Another benefit is redundancy, as a packet will be sent twice through different routes , and if packet loss occurs, it will be less likely to do on both packets.

Now I do not really know how complicated this can be to implement, but I can give it a try. So, what do you think?

[blast]

2.99.x had lag compensation, but it was designed backwards. It gave people with higher lag an advantage over those with lower lag. From what I recall, it affected at least bullets and possibly tank position. Let's say player A has 100ms lag and player B had 500ms lag. When player B fired a bullet, player A would see the bullet spawn 600ms in front of player B's tank, giving player A much less time to react (and possibly spawning the bullet inside of player A's tank).

We need to do it in such a way that low lag is rewarded and high lag is punished, not the other way around. And what really needs to happen is making the server be authoritative. Clients would then just send their input and receive relevant state from the server, and would no longer decide if they got shot.

Option 3 sounds like a nightmare with no benefit. You'd have different lag times, messages arriving in different orders from different sources, increased data rates. I just don't see how that could work, or why you'd want to do that.

[etigah]

The lag compensation method used in other games which you have described will not work for bzflag. And I think this is something all active players can agree on.

Rewarding based on latency is rewarding players for where they are located, which I think is not the way to go. The solution should improve game play for everyone.

For option 3, as I said, I haven't tried to work it in code, but in theory, players in the same region will always experience low latency as if played on the closest server. Different lag times are predictable based on geo location. I do not think data rate is an issue since the game already uses too little BW. Also the order of events is irrelevant I assume, only first handled event is considered , any other with same id is dropped.

[macsforme]

2.99.x had lag compensation, but it was designed backwards. It gave people with higher lag an advantage over those with lower lag. From what I recall, it affected at least bullets and possibly tank position. Let's say player A has 100ms lag and player B had 500ms lag. When player B fired a bullet, player A would see the bullet spawn 600ms in front of player B's tank, giving player A much less time to react (and possibly spawning the bullet inside of player A's tank).

I'm not sure how else you would "compensate" for lag, except one of:

1) push shots forward in time on clients other than the one shooting, as you described, or
2) have all hit detection done server-side and force clients to "lag dodge" relative to their own lag, or
3) make each client delay firing of their own shots for an appropriate period of time

Furthermore, in your example (at least theoretically... I'm not sure how it was implemented), a bullet fired by player A would also show up on player B's screen 600ms in advance, so it shouldn't be an advantage to either player more than the other. It is, nonetheless, a solution with pitfalls.

etigah, I've pondered the idea of a "lag ghost," which would be some indicator of where you actually should fire to hit a tank (this would, of course, jump around quite a bit when a laggy player changes directions). This should probably be an "advanced" setting for people to enable only if they understand it.

I don't like the idea of adjusting shot speed based on lag. If you consider just the lag between the two players, it would take the entire trip of the shot for the compensation to be realized. If you also factor in the distance between the two players, some shots would have to be extremely fast if the distance is small (which, if the player jumps away or something, would look very silly if they miss). Also, since compensation is done between each two players differently, shots that are fired at and hit another player might look to a third party like they were nowhere close to making impact.

[blast]

The problems we have are not unique. There's plenty of information about how to do proper game design now. For instance:
http://gafferongames.com/networked-phys ... -lockstep/ (and the other articles on the right side there)
http://www.runequake.com/hoh/Quake.pdf
https://developer.valvesoftware.com/wik ... timization

I can't recall how the lag comp in 2.99.x worked exactly now. Clients had a synced clock that was lag comped (I think), and the shot start time was synced to that clock. The start time of the shot was recorded by the client firing and encoded into the shot message. I believe in practice that it meant that the higher the shooters lag the further in front of their tank the bullet "spawned" for other players. It allowed someone with high lag to jump up at someone on a box and spawn a bullet inside their tank.

[etigah]

Constitution
Consider we have players A,B and C , and they lag 100, 200, 300 respectively. When player A shoots, player B will see the bullet moving at 1.1 times normal speed, and player C will see it moving at 1.2 times normal speed. It does not depend on the position of any player, but only their combined lag.

blast
I think BZFlag problems are unique. Unlike many other games, client side collision detection is the only possible approach, otherwise the game will be unplayable. And player movement is meant to be very precisely replicated, and can not be meaningfully predicted based on current player speed/position. The approach I suggested does not try to do any prediction while also not compromising the client side collision mechanism.

[blast]

What do you mean by client side collision detection? There would still be client side collision detection. There would just not be death calcs. The client would continue to run a copy of the simulation. But the server would also run this, with more historical information stored, and in the event that the client and server disagree, the server wins. Players would no longer send "I am here moving at X speed and turning Y fast", they would simply send a relative movement direction with some fixed increments, and both their local simulation and the server simulation would figure out the right speeds. Again, this isn't unique to BZFlag.

[etigah]

I meant that death calc should continue to be done based on client side detection, having it any other way will adversely affect enjoyment of the game.

BTW, I think that bullet speed increase is not very different from spawning a bullet in a more advanced position, the only difference being that in case of altering bullet speed, the change is gradual.

Perhaps changing bullet speed for a small period then restoring to normal would have the advantage of both methods?

[macsforme]

The problems we have are not unique. There's plenty of information about how to do proper game design now. For instance:
http://gafferongames.com/networked-phys ... -lockstep/ (and the other articles on the right side there)
http://www.runequake.com/hoh/Quake.pdf
https://developer.valvesoftware.com/wik ... timization

The one article of the three you linked that actually refers to hit detection is for instant-hit weapons, which does not apply to BZFlag unless you want to make every shot a laser bolt. I agree with etigah that our situation is a bit unique. Every solution I've considered also has negative trade-offs.

[blast]

So you take one of their simplified examples about the problems with predicting movement and weapon effects client-side and use that as an excuse to discount the entire article? You do realize that both Quake 3 and Half Life have weapons that aren't instant-hit, right?

[etigah]

blast, I have read the documents you shared, but I have not found any thing that change my mind on the subject. The methods applied in these cases if taken to BZFlag will lead to either or both of:
1- Jerky movement on the client, as the server will often disagree with the client and the client has to adjust to the server state
2- Unexpected deaths that can not be explained by looking at the radar representation of the map

The use case for BZFlag is different in that:
1- A player can clearly see the shot path and react in time. While other games, even those where some weapons are slow, player is not expected to rely on watching the shot as it comes.
2- Moving a very small distance, a few pixels, makes a great difference in outcomes, so jerky movement is not acceptable

The solution I suggest does not suffer from any of these issues. Sure it may have its own shortcomings, so I will be happy to get any help testing it. I forked the BZFlag repo on github and pushed my changes. And there is a test server running right now.

[Exuro]

etigah, if I understand you correctly:

5 players (each with different lag) simultaneous shooting at me would mean I see 5 shots coming at my tank, each with a different speed?

[etigah]

Exuro, yes, this is how it works. In reality the variation is not too great, as it fluctuates between 0% for very low lg and around 8% for very high lag (500ms combined)

[Exuro]

So, different bullet speeds eventually mean (when shot at):

# you either can travel a specific distance while the bullet is on its way or you can not.
# you either can turn around a certain angular degree or you can not
# you either can dodge a shot or not (because it is distance/ bullet speed depending)
# you have enough airtime (after jumping) for a specific action or not
# ...
... depending on from whom the bullets are.

This all varies depending on who shoots at you - so you will have to learn a lot (of those different lag-combinations) and this only works if the bullets are marked with "speed"indicators (and not the tanks, like you suggested).
You would have to mark every bullet with a speed indicator.

[etigah]

There are two possibilities
- Mark tank differently based on its current lag. The result is that bullet speed from the same tank can vary if there is a lag spike
- Fix the bullet speed for every tank based on long term average lag, ignoring any in-game lag variation

[Exuro]

You still have the problem that players have to adjust to a multitude of different bullet speeds.

Marking the tank does not help any bit when you see the bullets in your HUD or on radar, you can't be sure which of the other tanks shot at you.
So you really would have to mark the bullets.

[etigah]

The tank indicator would be visible on the radar. I do not see why it would be hard to identify the source of bullets in this case. Perhaps in a situation where there are 4 or more attacking tanks this would be an issue, but this is a rare situation.

[Zehra]

It seems like it may solve issues while creating them.
We're all used to the bullets traveling at a certain speed, but if we have to a guess how fast a bullet will travel it will throw for a while dodging and timing off.
The main goal would be to see if the server will agree with the clients on moves on whether the tank is hit or not.(See https://en.wikipedia.org/wiki/Benchmark_(computing).)

-Zehra

[etigah]

Actually we are used to a lot of variation, since tank velocity is added when determining shot speed. If my understanding is correct, a tank moves forward at 25 world units/sec, and backward at 12.5 wu/sec which means the slowest shot is 87.5 wu/sec and the fastest is 125 wu/sec . For this mod to have significant effect, the increase in speed is no more than 8% of the original

[Zehra]

Actually we are used to a lot of variation, since tank velocity is added when determining shot speed. If my understanding is correct, a tank moves forward at 25 world units/sec, and backward at 12.5 wu/sec which means the slowest shot is 87.5 wu/sec and the fastest is 125 wu/sec . For this mod to have significant effect, the increase in speed is no more than 8% of the original

Now if I remember correctly the tank is around 2.80 units wide and approximately 8 units long and around 2 units high.
And if my calculations would be correct the movement is from the tank is around 15.6 times the length of the tank per second.
Which means that the timing has to be within 1/16 of a second at the right position for a correct hit.(and this is by length not by with of the tank.)
Around 5% of change in speed means that it will be it will be around 7.25 world units change in movement and that could make it harder to decide if there could be hit or miss on target due to having to compensate for the modifications that may or may not occur.
I think this needs a benchmark test to eliminate randomness in a test and to be certain that it would improve the game.
And that the server and client have very little if any conflict on decisions on weather a tank is hit if a server decides that tank is hit.(If it server deciding on weather a user is hit gets implemented.)

[etigah]

I consider this suggestion rejected. Thanks for your time.

[click click boom]

Actually I don't mind this idea. But I would only want the plugin to work if the combined lag is high like over 450. I think this idea would be best on a mofo server rather than LU. Good job etigah.

[etigah]

ccb, I have concluded that the change to game play introduced by altering bullets speed would throw many LU players off. Perhaps if it is a new game, or as you said, on other maps, this might fit better.

IMO, that leaves us with the only other mean to improve the situation, that is showing the lag stats on radar.

[alfa1]

I have in general 200 or 300 ms of lag, depending if I am playing in NA or EU (I am from SA). As BZFlag is a complex game, IMO, adding still more info would make it more complex to manage, remembering that it is also an arcade game, which requires speed for reactions/thoughts (it is like a simulator + arcade game). In my case, I am in general the most lagger of any map (I play all styles), but sometimes I have mates from Oceania which are 50 ms more than me (250/350 ms). The suggested common solution is hunting them, then when I have them in front of me I receive a sound alert, the name of the player on top (I need it because I can be hunting others for other reasons) and, in radar, I always have their blinking colors showing. This will make me predict their movements and shooting ahead (the same method is for those who need to hit me, 'cause I am a lagger for them). I want to say too that having a high lag is also hard for laggers; in practice, it gives us less time to react, though, in the other hand, we get used to shoot low laggers in general. What I want to highlight, then, are 2 things:

first) the hunting method is nowadays simple, easy and enough successful, let's say
second) the lag compensation approach needs to be studied very well because it is rather complex and there must be different solutions (if there is a real good one at the end)

I don't like to talk if I have not read enough information but, it seems to me, as an experienced player, that making the shot speed variable would modify too much the playstyle. I would not drop it as a solution, but it would need enough study and experimentation.

Seemed a nice discussion, I can see good minds around .