The whole sample doesn’t matter, if the matchmaking data includes SU-5 and A-20 (vehicles with completely separate matchmaking rules) together with “regular” vehicles, it disqualifies the entire dataset as you can’t really come to any conclusions based on the mish-mash data.
I could claim that 90% of my games I am in top 5 and “forget” to mention that vast majority of the time I am playing IS-3, with KV-1S and Stug thrown in here and there.
([edit] I re-read your post and I think your point was that maybe Tman didn’t include SU-5 and A-20 into his sample? But then why would he state in that very post that’s those are the vehicles he is driving atm?)
There are more problems with his data - for example, the way he breaks down the placement into 5 groups (1-3, 4-6, 7-8, 9-12, 13-15).
- Why the middle group includes only 2 positions and the next one includes 4? To inflate the numbers related to lower positions?
- What does alphabetical placement on the list have to do with balance? On my IS-3, for example, I am quite often placed in 5-6 position but only because there are 4-5 tanks of the same tier above me alphabetically. Or the other way around when bottom 5 tanks are of the same tier.
- If some arty is placed in the 12th position, pushing me into the 13th, what does it matter? Am I somehow in a worse position than the same tier tank that’s been placed above the arty?
A more correct way to gather statistics would be to count tier differential. Because that’s what we are interested in, right? Not some arbitrary position on some arbitrary list. Are you 0, -1, -2, -3 or -4 to the top tier tank? Do it for 100 matches on the same tank, summarize, divide by 100 and look at the result. If it’s 3.5 then yeah, the game is fucking with you.
And make sure all your 100 matches are played in a similar “environment”. You have to compare apples to apples and games played on Saturday afternoon when 5x exp promotion is going on are going to be quite different from the games played at 2 am on Wednesday.
And do it within reasonable time - if it takes you a month to play these 100 games, your earlier results are too old and can’t be compared with the recent results (maybe they are tweaking the matchmaker?).
And then do it for every tier and every class. Because who knows, maybe the matchmaker is screwed up for tier 3 but works fine for tier 4.
And even then your analysis would be of so-so reliability because the sample is too small and the outside parameters are not defined and not fixed.
Statistics is a funny science. It’s hard enough to analyze the data that you do have (like the WoT devs do), it’s a million times harder to come up with reliable data to perform some “homemade” statistics analysis.