Go Back   Star Trek Online > Feedback > Duty Officer System and R&D
Login

Reply
 
Thread Tools Display Modes
Captain
Join Date: Aug 2012
Posts: 3,677
# 21
02-06-2013, 03:03 PM
Quote:
Originally Posted by kagasensei View Post
A good first-order estimate for a sufficient sample size is always the square of the number of elements --> 42 consoles --> 1764 = N

way to go
In general statistics though, 10,000 is usually the accepted minimum accurate sample size. It gives enough room for variance and enough room for outliers to be negated by a standard curve. Since I don't feel like thinking atm, I just shot that out since my stats professors slammed that number into me since college.
It is said the best weapon is one that is never fired. I disagree. The best weapon is one you only have to fire... once.
Why the Devs can't make PvE content harder. <--- DR proved me wrong!
Empire Veteran
Join Date: Jun 2012
Posts: 2,202
# 22
02-06-2013, 03:10 PM
Quote:
Originally Posted by borticuscryptic View Post
The only possible explanation I can offer at this point, is that it's just a matter of sample size. There are 42 different Mk XII consoles you can receive from this assignment (16 Eng, 10 Sci, 16 Tac) of each quality (Green, Blue, Purple) resulting in the total number of potential outcomes being 42*3=126. When you run a sample size of 300, with almost half that many possible outcomes, your odds are invariably going to be too small to create an big enough picture of the entirety.
With such a number of consoles you need ~10k attempts to get a reliable sample.

FYI i must have been lucky, half of the time i get something useful. Last week, i got a purple monotanium alloy and phaser relay, and a blue shield capacity console. I also had a chroniton console. My guess is that only unlucky people will complain, happy ones don't feel the need to rant on the forums.

Last edited by diogene0; 02-06-2013 at 03:14 PM.
Captain
Join Date: Aug 2012
Posts: 3,677
# 23
02-06-2013, 03:21 PM
Quote:
Originally Posted by diogene0 View Post
My guess is that only unlucky people will complain, happy ones don't feel the need to rant on the forums.
This statement combined with your profile picture had me laughing so hard...
It is said the best weapon is one that is never fired. I disagree. The best weapon is one you only have to fire... once.
Why the Devs can't make PvE content harder. <--- DR proved me wrong!
Starfleet Veteran
Join Date: Jun 2012
Posts: 716
# 24
02-06-2013, 03:33 PM
I don't think many people have a concept of what "random" or "statistical chances" actually mean.

Take a six-sided die. It has... six sides. If you roll it, it will result in one of those six sides facing up. Each side statistically has a 16.6666666666667% chance of landing face-up.

However, this does NOT mean that a particular side will come up one out of six times. It only means that each time you make a roll, each side has an equal 16.6666666667% chance of landing.

If you roll a "1", you will not necessarily get a "1" every six times. You're not going to roll the die six times and be guaranteed a "1". If you roll the die 36 times, you're not going to get equal results for each side. Even if you roll the die a billion times, the results will not be even across all possible results despite each side having an equal chance of landing.

Random is random. A probability statistic only matters to the individual instance, not a group of the same action being performed over and over. If you get a particular result with one instance, it does not change the probability of a different result in a different instance.

(Ex: if you get a "1" result on a six-sided die, it does not mean that you are any more or less likely to get that same or different result on successive rolls)

Raptr profile
Lt. Commander
Join Date: Jun 2012
Posts: 215
Starfleet Veteran
Join Date: Jun 2012
Posts: 197
# 26
02-06-2013, 04:37 PM
Quote:
Originally Posted by tehjonel View Post
I'd love this, if it weren't so true

Seriously though, that's funny.
Starships: Model errors and feedback project, 2410 edition.
Career Officer
Join Date: Jun 2012
Posts: 464
# 27
02-06-2013, 05:28 PM
Quote:
Originally Posted by borticuscryptic View Post
I found these results interesting enough that I decided to dig into the reward structure of console fabrication. Since these were created under Heretic, I truly didn't know what to expect.

Here are my findings:

1) The quality of the reward outcome of the "Fabricate Prototype" assignment is guaranteed to match the quality of the success' display name. In other words, if your results say "Blue Quality" the console you receive cannot be anything other than Blue (Rare) quality, in all circumstances. No rare chance to upgrade to a better type, or anything like that.

2) Every console available from our random drop tables is represented. Despite the above sample showing zero Phaser Relays (e.g.), they are included in the drop table as a possible result.

3) All console rewards are equally weighted. This is the part I was unsure would be true, but it is - every item on the reward tables has an absolutely equal chance of being rewarded.

The only possible explanation I can offer at this point, is that it's just a matter of sample size. There are 42 different Mk XII consoles you can receive from this assignment (16 Eng, 10 Sci, 16 Tac) of each quality (Green, Blue, Purple) resulting in the total number of potential outcomes being 42*3=126. When you run a sample size of 300, with almost half that many possible outcomes, your odds are invariably going to be too small to create an big enough picture of the entirety.
There is something definitely amiss and while everyone is pointing to sample size issues and "statistically speaking" I can defiantly tell you it is not a sample size issue.

Let us forgo the rarity factor because the op stated those proportion came out the same and were find. Let?s focus on the fact that each console type is not dropping in ?equal? proportion. Is a sample size of 300 sufficient? In statistics, we have two error rates, alpha is the probability of making a type I error (incorrectly rejecting a true null hypothesis) and beta is the probability of making a type II error (failing to reject a false null hypothesis). You only worry about the type II error rate when you do not have a significant alpha (usually >0.05 in most hypothesis testing). When you reject a null hypothesis you have to check is you had enough power (beta and sample size) to ensure you did not make a false rejection.

Now with this primer behind us let us do a simple and quite easy to do statistical test. The type of test that fits this data the best is a chi-square test. For this test, we take our observed values (from the op) and derive expected values. In this case, we know the consoles should drop in equal proportion so the number of expected consoles of a particular type say for science is simply 1/10 = 0.1 * Observed value. The OP had 131science consoles so the expected values following a uniform distribution is 13.1 consoles. He should have received 13.1 of each particular console type (if they drop uniformly). Now for those of you that want to look up the chi-square test there are plenty of hits on Google. Now we have our null hypothesis that says the observed drops = the expected drops and the alternative hypothesis that says the observed drops do not equal the expected drops. I am going to go to the next step.

For the science consoles we get a chi-square value of 31.8, with 9 df, and a probability = 0.000214. Thus, we reject the null hypothesis and the observed drops do not equal the expected drops. Because our alpha is <<0.05 we do not have to worry about a beta error rate or our sample size because the effect is strong.

For the engineering consoles we get a chi-square value of 136.0, with 14 df, and a probability = 4.36x10^-22. Thus, we reject the null hypothesis and the observed drops do not equal the expected drops. Because our alpha is <<0.05 we do not have to worry about a beta error rate or our sample size because the effect is strong. By the way I could only find 15 not 16 engineering consoles.

For the tactical consoles we get a chi-square value of 72.36, with 15 df, and a probability = 1.69x10^-09. Thus, we reject the null hypothesis and the observed drops do not equal the expected drops. Because our alpha is <<0.05 we do not have to worry about a beta error rate or our sample size because the effect is strong.

Thus, sample size is not the issue here because the effect or magnitude of the difference is so strong. Our conclusions are then within each console group, a particular console type is not dropping uniformly. I have my suspicions it is the way the reward rolls and loot tables are structured. Instead of rolling for rarity, then console group, then console type the system is set to roll first for rarity then console type. All the consoles are pooled and we can now test that hypothesis. So our expected number for a particular console type is 1/41 = 0.02439 * Observed Value which means the op should have received 7.65 of each console type if again they drop uniformly.

Running our chi-square analysis again we get a chi-square value of 263.39, with 40 df, and a probability = 1.14x10^-34. Thus, we reject the null hypothesis and the observed drops do not equal the expected drops. Because our alpha is <<0.05 we do not have to worry about a beta error rate or our sample size because the effect is strong. We can say that the consoles do not drop uniformly when they are pooled.

So we can say statistically consoles from the DOFF crafting mission are not dropping uniformly within console type or within the pooled subset. We can also say that with great certainty that the sample size is not an issue because the magnitude of the effects are so great. How is the drop rate then calculated? I cannot answer that, all we can say it is not uniform based on the two scenarios above. More likely the probability of getting a specific console of a specific quality using the OP?s quality rates are:

All Consoles Pooled
Very Rare ? 0.24*0.02439 = 0.005854 = 0.59%
Rare ? 0.51*0.02439 = 0.012439 = 1.24%
Uncommon ? 0.24*0.02439 = 0.005854 = 0.59%
If the OP can provide a breakdown of the qualities we can even go so far as to test that.
Captain
Join Date: Jun 2012
Posts: 3,215
# 28
02-06-2013, 07:48 PM
i just realized: 40% of all possible consoles are worthless ingame...srsly worthless. compareable to cloth armor items with strength as main attribute in other MMO's.
Go pro or go home
Career Officer
Join Date: Jun 2012
Posts: 464
# 29
02-06-2013, 10:26 PM
Quote:
Originally Posted by hereticknight085 View Post
In general statistics though, 10,000 is usually the accepted minimum accurate sample size. It gives enough room for variance and enough room for outliers to be negated by a standard curve. Since I don't feel like thinking atm, I just shot that out since my stats professors slammed that number into me since college.
I believe you may have mis-interpreted things or the last hundred years of scientific investigation is rendered null and void by that stat professor of yours. I believe what he was talking about was re-sampling statistics such as bootstrapping, jack-knifing, Markovian-chains etc... The general accepted rule for those is a minimum of 10,000 replicates/resamples but with more powerful cumputers today 100,000 replicates are easy. In addition, since we are talking about count based data here and not continuous scale data, measures of central tendency do not really apply.

Last edited by commodoreshrvk; 02-06-2013 at 11:03 PM.
Career Officer
Join Date: Jun 2012
Posts: 464
# 30
02-06-2013, 10:39 PM
Quote:
Originally Posted by kagasensei View Post
A good first-order estimate for a sufficient sample size is always the square of the number of elements --> 42 consoles --> 1764 = N

way to go
I can honestly say that in all my years of teaching the people that now teach stats and all the stat work that I do, I have never heard of such a "general rule". Do you have a citation for this because I would be very interested in reading it? I have heard all sorts of things though such as having a sample size 15 times the number of variables to be estimated. Your best bet is to conduct an a priori power analysis but you would need a pre-determined effect size. Also, for the example and discussion here what you are really testing or wanting to examine is IF the data fit a pre-determined distribution. In this case, that is a uniform distribution (all have an equal probability of dropping).

At some point the sample size may even give a difference that only has a statistical meaning. This is not the case when distribution testing or fitting as the greater the sample size the better the fit or lack thereof.

The point is it is quite obvious that the console drops fit a logarithmic or inverse hyberbolic distribution whether that is intended or not.

Last edited by commodoreshrvk; 02-06-2013 at 11:02 PM.
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


All times are GMT -7. The time now is 12:27 AM.