Rating and rankingsystem

What's the best solution? Discuss...

Discussion about the game and its default mods.
noobmapmaker
Posts: 1086
Joined: Wed Dec 10, 2014 11:59 am

Rating and rankingsystem

Post by noobmapmaker »

We're going offtopic in the other thread, and I think this deserves it's own topic.

So will a rating and ranking system be an addition to this game and the community?
If so: what should the ranking system do, and what not?

Do you know any good examples from other communities?

Questions that need answers:
- does the nature of this game (RTS) allow ranking? How does Starcraft address this?
- what are the downsides of a rankingsystem, can they be addressed?
- what are the upsides?
- is this community ready for ranking an rating?
- are there other solutions that do not need widespread implementing of code, e.g. an external website to organise competitive play?

I'll gather some information overtime and post it here.
Last edited by noobmapmaker on Wed Jun 10, 2015 11:14 am, edited 2 times in total.

noobmapmaker
Posts: 1086
Joined: Wed Dec 10, 2014 11:59 am

Post by noobmapmaker »

A gamingsite that has a nice implementation is www.asobrain.com

They have a yearly ranking and a monthly ranking. So each month players compete to be the month's nr 1 player and meanwhile all their games add up for their yearly ranking.

See:
Their ranking pages (can be sorted per month/year):
http://games.asobrain.com/pages/ranking/index.php
Ranking list for Cities and Knights:
http://games.asobrain.com/pages/ranking ... nkingID=11
Their FAQ on ranking:
http://games.asobrain.com/pages/faq.php?section=ranking

The website mainly features Settlers of Catan, a game where players gather points and win when they reach the set goal. One can win big or small. This is different from chess or OpenRA as players then only win or lose (big or small loss is not taken into account).
Last edited by noobmapmaker on Wed Jun 10, 2015 10:58 am, edited 1 time in total.

noobmapmaker
Posts: 1086
Joined: Wed Dec 10, 2014 11:59 am

Post by noobmapmaker »

Simple explanation of ELO: (from Reddit)
While there has been criticism of using Elo (a system developed for 1 vs. 1 play) for team games, where a player's success can depend on other players, time and time again it's proven to accurately reflect an individual player's skill over the long term.
Here is something key to understanding Elo – Elo is not a system you can "grind" to get to the top
You are not guaranteed to go up if you keep playing. Elo is a system that places you where you belong, by your skill, in relation to other players. There is an endless amount of evidence verifying the accuracy and efficacy of the system.
Elo works off 2 primary numbers, your Elo score and your tier. You tier determines your "kFactor", which is used in the calculations behind Elo.
kFactor determines how much you should move around in the rankings. It's very high when you start, to allow for you to be placed in the right tier after playing a few games, and lowers over time and performance, so you will not be affected greatly by one win or loss after playing many matches.
In other words, when you start playing or are in a low tier, expect big changes to your Elo. When you are in a high tier, small changes to your Elo are common.
More ELO info from Wikipedia: http://en.wikipedia.org/wiki/Elo_rating_system

From that wiki also see: (possible dangers/disadvantages)
- Selective pairing
A more subtle issue is related to pairing. When players can choose their own opponents, they can choose opponents with minimal risk of losing, and maximum reward for winning.
- Game activity versus protecting one's rating
In some cases the rating system can discourage game activity for players who wish to protect their rating.[22] In order to discourage players from sitting on a high rating, a recent proposal by British Grandmaster John Nunn for choosing qualifiers to the chess world championship included an activity bonus, to be combined with the rating.[23]

Beyond the chess world, concerns over players avoiding competitive play to protect their ratings caused Wizards of the Coast to abandon the Elo system for Magic: the Gathering tournaments in favour of a system of their own devising called "Planeswalker Points".[link to site explaining that system][25]
Another interesting system is the Glickosystem, that is used by chess.com.
Read more about it here: http://www.glicko.net/glicko.html
The Glicko system therefore extends the Elo system by computing not only a rating, which can be thought of as a “best guess” of one’s playing strength, but also a “ratings deviation” (RD) or, in statistical terminology, a standard deviation, which measures the uncertainty in a rating (high RD’s correspond to unreliable ratings). A high RD indicates that a player may not be competing frequently or that a player has only competed in a small number of tournament games. A low RD indicates that a player competes frequently.

In the Glicko system, a player’s rating changes only from game outcomes, but his/her RD changes both from game outcomes and also from the passage of time when not playing. One feature of the system is that game outcomes always decrease a player’s RD, and that time passing without competing in rated games always increases a player’s RD. The reason is that the more games played, the more information is learned about a player’s ability, so the more precise the rating becomes. As time passes, we become more uncertain about the player’s strength, so this is reflected in the RD increasing. It is interesting to note that, in the Glicko system, rating changes are not balanced as they usually are in the Elo system. If one player’s rating increases by x, the opponent’s rating does not usually decrease by x as in the Elo system. In fact, in the Glicko system, the amount by which the opponent’s rating decreases is governed by both players’ RD’s.

noobmapmaker
Posts: 1086
Joined: Wed Dec 10, 2014 11:59 am

Post by noobmapmaker »

Some views from the other thread:

Raymundo:
I don't think "rank" is important for this type of game. Rank won't be a great indicator of skill because this is primarily a team based game. Once you introduce that system into team game, I believe problems would arise. Some could be that players would be even more unmotivated to play a match if they have to play with someone who they know is less skilled, or your teammates could easily affect your rank if they are distracted or afk, etc. So if other players have such a big factor on your rank, it doesn't really show how skilled you are anyway. The only type of ranked match I think would work are 1v1's. Possibly a check box could be in match options to turn ranked on or off. This could have a negative impact also though, maybe a big focus of the game will become 1v1 matches and team based games which are where this game really shines would decrease in popularity.

I think all we need is an indicator of skill, so people know what they're up against. I may be alone on this but I don't think the fun of this game would be competing for the top spot, it comes from having interesting and challenging matches. Solution - tie number of games a player has played to their IP, so even when they play on a different *registered* name they will appear as having experience playing the game. Of course having played 200 matches doesn't mean your the best, only that other players would expect that player to be competent.

I think rank should come from an organized ladder, and seasons which could be done outside of the game on tournament websites. If the site and the openra dev are able to coordinate players could receive a sort of emblem next to their name in lobby for achieving high rank in these tournaments.

Summary :
- Need registered names
- If ranked matches are added let them be for 1v1 only
- Tie ranked tournaments or ladders from an outside website into the game and let that be used for rank. Ex: your 2v2 team wins a seasonal tournament and you both receive a certain emblem to be displayed next to your name in lobby.
Murto the Ray:
I'd just like to see some kind of indicator of matches played to serve as an indicator of how new or experienced someone is - single player doesn't count
Winftw
A ranked ladder would be awesome. Some might argue we need a huge playerbase for a such thing but IMO a ladder is actually a prerequisite for a huge playerbase.
Zoidyberg:
I can't see how a ranking system would benefit the OpenRA community, even if it could be properly implemented. Just enjoy the game.

OpenRA. Players will gradually cluster together into trusted groups because they want to increase or maintain their coveted rankings. Games are already sparse and a scoring systems will create an artificial barrier that should not exist.

Let's say the decision is made to introduce scoring. The scoring process will need to be refined for some time because players will "game the system" and devalue the rankings. For every hour a developer spends on dealing with players who cheat the scoring system, that's one less hour that can be spent on things like game balancing, networking issues etc.
AoAgeneral1
While a scoring system is a good idea it can also seperate the community. Luckily for me, ive been able to play against anyone who joins in (Unless they know the name then they jump back out). If the match making system is involved it can create staggering game issues as it did with Grey Goo. Trying to match players equal level only the system can't find enough players to match with and gets matched with a strong vs new player.

I prefer the system as it is right now and just work on the network bugs and issues. We have spectator problems where if a player leaves ingame while spectating the replay breaks.
Spectating a lobby can prevent other players from joining. and etc.
Some of the things that should be ironed out first before a ranking system comes into mark. Ranking system has its pros and cons.

winftw
Posts: 25
Joined: Tue May 05, 2015 10:24 am

Post by winftw »

My point of view is that I want to get ingame asap with one click. Thats what a quickmatch ladder basically does and the reason why I think it could help openra get more players. Getting the skill level right isn't that important to me but just icing on the cake.

Anyway I totally agree with everyone that a rating system can divide community and encourage cheating and bad stuff.

The whole "gaming the system" problem could be avoided by having player ratings be secret. Only the matchmaking server really needs to know player ratings in order to set up games.

Secret player ratings could also be used for other kinds of games and not just 1v1 ladder games. Example:
1. a bunch of players login to a ratings server with their accounts and then make/join a game
2. the game sends a list of players to the ratings server
3. the ratings server reports back how to setup balanced teams
(the match is played out)
4. match outcome is sent to the ratings server which then updates elo/whatever

Having unranked games alongside is also a must because people want to play with their friends etc. And maybe the ratings system ends up being broken anyway so we could fall back to the tried and true method.

Murto the Ray
Posts: 486
Joined: Mon Nov 10, 2014 4:34 pm

Post by Murto the Ray »

I think the main problem, which has been brought up several times already, is that OpenRA doesn't have the mass community that a game such as StarCraft does. As has already been mentioned, this also means times when barely any players are online (i.e. 0600-1000 GMT) there will be very little fair match ups.

Despite these points i believe a ranking system should be implemented into OpenRA at some point, however for now we should focus on introducing more players into the newbie friendly team games instead of having them come into a harsh 1v1 environment where laddering is essential to being reputable or deemed useful in a custom team game.

User avatar
raymundo
Posts: 117
Joined: Mon Mar 30, 2015 10:39 am

Post by raymundo »

No matter what happens, the first step will have to be registering names, so let's see that implemented first. I personally don't care about "rank" and I don't see why anything has to be ranked in game, I play this game for fun and challenging matches. We are able to coordinate tournaments outside the game, maybe we could expand on the battlefield news to have information on tournaments and standings.

noobmapmaker
Posts: 1086
Joined: Wed Dec 10, 2014 11:59 am

Post by noobmapmaker »

winftw wrote: My point of view is that I want to get ingame asap with one click. Thats what a quickmatch ladder basically does and the reason why I think it could help openra get more players. Getting the skill level right isn't that important to me but just icing on the cake.

Anyway I totally agree with everyone that a rating system can divide community and encourage cheating and bad stuff.

The whole "gaming the system" problem could be avoided by having player ratings be secret. Only the matchmaking server really needs to know player ratings in order to set up games.

Secret player ratings could also be used for other kinds of games and not just 1v1 ladder games. Example:
1. a bunch of players login to a ratings server with their accounts and then make/join a game
2. the game sends a list of players to the ratings server
3. the ratings server reports back how to setup balanced teams
(the match is played out)
4. match outcome is sent to the ratings server which then updates elo/whatever

Having unranked games alongside is also a must because people want to play with their friends etc. And maybe the ratings system ends up being broken anyway so we could fall back to the tried and true method.
Good post, I also like the secret rating as it prevents people from not playing because they might lose a spot or go under X-rating points. It would be interesting to publish the monthly ranking though (without ratingpoints, just rank) so you know where you stand.

If the ratings are secret, then in my opinion any game can be ranked, even those with friends. The algorithm then does need a line that prevents pointsgrinding against people. (really sad if that is what you want to do, but people do it)

What I like about your setup is that people can join the server/map they want to, then the admin presses "Balance Teams" en *pooof* there are balanced teams (as good as possible). This system is in my opinion fun for newbies as they will team up with a stronger player. It allows them to learn and play a game that is not instantly lost like it would be when they end up with another new player.

Balancing does have an issue, that is mentioned with several systems:
How do players compare? When is a player twice as good as another? Is that 1000 + 1000 vs a 2000 or is that 1500 + 1500 vs 2000? Is a 1800 + 2000 vs 1900 + 1900 a balanced match? Hard to tell on forehand as we dont know how the ratingpoints reflect skill(difference), yet.

Also agree with others that first other hurdles, like registration, have to be taken.

zoidyberg
Posts: 90
Joined: Fri Dec 12, 2014 1:14 am

Post by zoidyberg »

noobmapmaker - Care to address any of the opposing viewpoints on this issue, discuss which problem a ranking system will resolve, or explain how implementing a ranking system would be a considered a goal-directed behavior. There may be a valid case for this, but I haven't heard it yet.

This entire conversation is the equivalent of somebody dragging the development team into a meeting to discuss a product enhancement which they would find be interesting. There has been zero regard given to a ranking system's purpose, goal, prioritization, or the required resources to implement and continually refine.

Z

User avatar
BaronOfStuff
Posts: 438
Joined: Sun May 22, 2011 7:25 pm

Post by BaronOfStuff »

Don't do this, or at the very least don't make any aspect of the stats public; the last thing a community like this really needs is a load of saddos berating new/casual players who don't sit at the PC all day replaying maps and learning build orders ASAP with endless chorus-calls of "noob" etc. while they and their pro MLG buddies all have e-cock measuring contests.

Stat-padding & seal-clubbing will become all too common and the relatively small community we have will be divided almost immediately between so-called "pros" and apparent scrubs. Just keep it fun and simple with stat-free tournaments and let user reputations in the community take place over a load of numbers and other pointless e-stat bullshit.

noobmapmaker
Posts: 1086
Joined: Wed Dec 10, 2014 11:59 am

Post by noobmapmaker »

zoidyberg wrote: noobmapmaker - Care to address any of the opposing viewpoints on this issue, discuss which problem a ranking system will resolve, or explain how implementing a ranking system would be a considered a goal-directed behavior. There may be a valid case for this, but I haven't heard it yet.

This entire conversation is the equivalent of somebody dragging the development team into a meeting to discuss a product enhancement which they would find be interesting. There has been zero regard given to a ranking system's purpose, goal, prioritization, or the required resources to implement and continually refine.

Z
Yes, there are many good reasons. And keep in mind that I try to gather information, opinions and anything relevant so that in the end an informed decision can be made, which may be in fact to NOT have a rating system. But in the end I do hope there will be some kind of ratingsystem, for these reasons:

1) Competitive play is fun
Reaching a topspot or climbing a couple steps on the ladder is fun. Getting an utterly worthless trophy for being March nr. 1 is fun. The competitive element might attract good players from other (strategy) games. In general people enjoy competition, see how happy people can be when they win a competition at low levels, or when they become a top 100 player.

2) It allows us to balance games
Any game with more than 2 players can be autobalanced according to ratings. This is especially an interesting feature for newbies as it prevents very one-sided games. How many games have been boring, both for the "pro team" and the "noob team", because they where unbalanced. When the player history autobalances the teams we will at least have the most optimal configurations.

3) It allows us to play handicapped games
When there are very few players you can still play an exciting game. At this time it must be somewhat hard for the best players to find good opponents. With handicapped games the game is still exciting for both players, and the game can be rated as if the players where equal.

All these 3 reasons can have possible downsides. Those downsides can be as big or small as we speculate them to be. In the end I think it is actually more Newbie friendly and it will lead to more equal and thus more exciting games. And yes we are limited by the developers resources, but that is the case for everything in OpenRA.

zoidyberg
Posts: 90
Joined: Fri Dec 12, 2014 1:14 am

Post by zoidyberg »

noobmapmaker wrote: Yes, there are many good reasons. And keep in mind that I try to gather information, opinions and anything relevant so that in the end an informed decision can be made, which may be in fact to NOT have a rating system. But in the end I do hope there will be some kind of ratingsystem, for these reasons:

This is a disingenuous statement. You are advocating for the introduction of a ranking system and are downplaying drawbacks.We both have an agenda.


here we go....


noobmapmaker wrote: 1) Competitive play is fun
Reaching a topspot or climbing a couple steps on the ladder is fun. Getting an utterly worthless trophy for being March nr. 1 is fun.
The resources required to create, police, refine a ladder at the very least diminish this position (if not wipe it out). Also, some players would not find this to be fun.


noobmapmaker wrote: The competitive element might attract good players from other (strategy) games. In general people enjoy competition, see how happy people can be when they win a competition at low levels, or when they become a top 100 player.
Using a ranking system as a recruiting tool is logical. When deciding on which games to play, I have never considered a ladder to be a deciding factor. The opposing viewpoint is that a ladder will split the OpenRA community and lead to less engagement.


noobmapmaker wrote: 2) It allows us to balance games
Any game with more than 2 players can be autobalanced according to ratings. This is especially an interesting feature for newbies as it prevents very one-sided games. How many games have been boring, both for the "pro team" and the "noob team", because they where unbalanced. When the player history autobalances the teams we will at least have the most optimal configurations.
This is a valid point. If there were 1,000 active OpenRA players, there may be a genuine need for a basic rankings system to avoid noob vs. pro games. A simplified solution to noob vs. pro game balancing is to display # of games played as a proxy for skill (instead of an arbitrarily level #).


noobmapmaker wrote: 3) It allows us to play handicapped games
When there are very few players you can still play an exciting game. At this time it must be somewhat hard for the best players to find good opponents. With handicapped games the game is still exciting for both players, and the game can be rated as if the players where equal.

Handicapping (slower tanks?) makes it more difficult for newer players to become more skilled. Developing a feel for the mechanics of OpenRA (damage, build times, speed of units, etc.) is essential to becoming a better player. Playing a simplified version of OpenRA with a lower tech level is probably a better way to handicap a game because there are fewer variables to manage.

noobmapmaker
Posts: 1086
Joined: Wed Dec 10, 2014 11:59 am

Post by noobmapmaker »

This is a disingenuous statement. You are advocating for the introduction of a ranking system and are downplaying drawbacks.We both have an agenda.

here we go...
.
No agenda, just an opinion for what I think would be a really nice addition. And I don't think I downplay drawbacks. I've actually named a few here and in the previous topic. And for some of the possible dangers I estemate them to be close to zero. Not because I want a ratingsystem, but because I think they will not play a role as much as others predict.
The resources required to create, police, refine a ladder at the very least diminish this position (if not wipe it out). Also, some players would not find this to be fun.
Resources is always an issue, also technically this idea is still far away. Yet still I like tinking about a ratingsystem that would 1) be fun to have 2) supports the game (and not notably hampers the game).

For one, I think the system will not incite insane competitive play where people call newbies "noobs" or curse the hell out of them because they build the wrong stuff. I think with a secret ratingsystem people will mostly just carry on playing as we do now. And RTS strategy is a quite competitive game in itself: you have to destroy the other. People who really dislike competition might not choose for this game at all. So I question how many people will be scared away by a (secret) rating system. Also because there are many websites with various types of games where both types of players (casual and pro) co-exist just fine. Admittingly: the sites I refer to are not teamgames. But as long as you keep the ratings secret (i.e.: the rating itself doesn't become the primary goal) there will probably not be a whole lot more teamcursing then there is today.
Using a ranking system as a recruiting tool is logical. When deciding on which games to play, I have never considered a ladder to be a deciding factor. The opposing viewpoint is that a ladder will split the OpenRA community and lead to less engagement.
Neither have I and nor should it be. Just join games and when it is anything above 2 players you can use the "autobalance" option to optimize the teams. Then play the game, which is, probably, with these players, the most exciting game possible at the moment.
At some point however, if this community keeps growing, it is recommendable to have a proleague or something like that. Just because it creates the best and most tense games when there is something at stake (even when it's just silly internetpoints or fame).
This is a valid point. If there were 1,000 active OpenRA players, there may be a genuine need for a basic rankings system to avoid noob vs. pro games. A simplified solution to noob vs. pro game balancing is to display # of games played as a proxy for skill (instead of an arbitrarily level #).
Imo that stat is pretty useless to reflect skill. A rating needs just a couple games to allmost reflect skill (with an ELO system with K-factor). Gamecount needs a whole lot of games to be of any meaning, and even then... Is 10 good? 20? 50? I for example may have played 300 games but in my belief my skill is pretty stuck on a certain level. A good starcraft 2 player probably beats me after playing 5 games.
Handicapping (slower tanks?) makes it more difficult for newer players to become more skilled. Developing a feel for the mechanics of OpenRA (damage, build times, speed of units, etc.) is essential to becoming a better player. Playing a simplified version of OpenRA with a lower tech level is probably a better way to handicap a game because there are fewer variables to manage.
You're right that getting a feel for the mechanics is an important aspect of the game. Personally I think that either the better player should have units that do less damage or gains less money from ore. The latter is the least intrusive as it doesnt change much for the newbie player. Doing less damage (and therefor either you need more units or better micro/macro management) is more intrusive as a player might get the idea that "just a couple tanks can kill a whole bunch of tanks".

I dont think a huge deal of handicapped games will be played. It's mostly good for 1v1 or if you don't autobalance the teams. Perhaps there should be some alert when the game starts. "You're playing a handicapped game. Enemy fire does 23% less damage" (23% is a random number, but should be based on rating). (not sure if the engine allows that btw..)

zoidyberg
Posts: 90
Joined: Fri Dec 12, 2014 1:14 am

Post by zoidyberg »

noobmapmaker wrote: Resources is always an issue, also technically this idea is still far away. Yet still I like tinking about a ratingsystem that would 1) be fun to have 2) supports the game (and not notably hampers the game).
Just because you say it will be fun to have, doesn't mean it will be fun to have. If you want to say that you would enjoy a ranking system, that's fine.



noobmapmaker wrote: For one, I think the system will not incite insane competitive play where people call newbies "noobs" or curse the hell out of them because they build the wrong stuff. I think with a secret ratingsystem people will mostly just carry on playing as we do now. And RTS strategy is a quite competitive game in itself: you have to destroy the other. People who really dislike competition might not choose for this game at all. So I question how many people will be scared away by a (secret) rating system. Also because there are many websites with various types of games where both types of players (casual and pro) co-exist just fine. Admittingly: the sites I refer to are not teamgames. But as long as you keep the ratings secret (i.e.: the rating itself doesn't become the primary goal) there will probably not be a whole lot more teamcursing then there is today.
Huh? What are you trying to get at?




noobmapmaker wrote: Neither have I and nor should it be. Just join games and when it is anything above 2 players you can use the "autobalance" option to optimize the teams. Then play the game, which is, probably, with these players, the most exciting game possible at the moment. At some point however, if this community keeps growing, it is recommendable to have a proleague or something like that. Just because it creates the best and most tense games when there is something at stake (even when it's just silly internetpoints or fame).
So you agree with me. Thank you.



noobmapmaker wrote: Imo that stat(experience) is pretty useless to reflect skill. A rating needs just a couple games to allmost reflect skill (with an ELO system with K-factor). Gamecount needs a whole lot of games to be of any meaning, and even then... Is 10 good? 20? 50? I for example may have played 300 games but in my belief my skill is pretty stuck on a certain level. A good starcraft 2 player probably beats me after playing 5 games.
Experience is a poor indicator of skill? What??



noobmapmaker wrote: You're right that getting a feel for the mechanics is an important aspect of the game. Personally I think that either the better player should have units that do less damage or gains less money from ore. The latter is the least intrusive as it doesnt change much for the newbie player. Doing less damage (and therefor either you need more units or better micro/macro management) is more intrusive as a player might get the idea that "just a couple tanks can kill a whole bunch of tanks".
Handicapping games for newer players will encourage them to engage in tactics that are not feasible when matched against experienced players. If anything, this will delay a new player's growth because they'll have to relearn how units behave as they progress through handicapped levels. Starting on a minimal tech level will encourage new players to master units and will expedite the learning process.



noobmapmaker wrote: I dont think a huge deal of handicapped games will be played. It's mostly good for 1v1 or if you don't autobalance the teams. Perhaps there should be some alert when the game starts. "You're playing a handicapped game. Enemy fire does 23% less damage" (23% is a random number, but should be based on rating). (not sure if the engine allows that btw..)
Why would you want to implement handicapping if only a few handicapped games will be played? It doesn't make sense...

Canavusbis
Posts: 37
Joined: Thu May 21, 2015 10:58 pm

Post by Canavusbis »

Handicapping will probably be implemented as the devs and community have wanted it for a while, but ranking seems foolish and counter productive for a game with so few resources ($, time) and players. It also provides another source of headaches for the devs. Our master server already has enough problems, I can't see any of the senior devs wanting more network-related headaches. The server operators have been very generous already, I believe asking them to maintain another system is pushing the envelope.
Zoidyberg wrote:This entire conversation is the equivalent of somebody dragging the development team into a meeting to discuss a product enhancement which they would find be interesting. There has been zero regard given to a ranking system's purpose, goal, prioritization, or the required resources to implement and continually refine.
You're mistaken, however. There has been a large regard into ranking, and as the previous tournaments have suggested, there really isn't enough interest in order to preform the tiring amount of work required to implement such a complex system.

Competitive play can be done using tournaments and your own communities.

Post Reply