The transformation you referenced is advocated by DRC (Data Recognition Corporation). The justification appears to be a desire to linearize the p-value after it has been calculated using the normal distribution (I.e., the average number of wrong-to-right answer changes in a school or classroom is compared with the population average using the central limit theorem and the normal distribution).
DRC calls this a threat scale. The constant 1.086 was selected somewhat arbitrarily so certain values of the resulting scale conform to select p-values. I don't remember which scale values they selected for alignment, but you should be able to figure this out.
The main justification for the transformation is that in test security work p-values need to be very small for flagging. There is a desire to be extremely conservative because these data were not collected according to an experiment and the underlying distributions have not been rigorously documented. Indeed, the distributions depend upon student ages (I.e., grade), the subject being taught, the amount of time allowed to take the test, and many other factors, including item-by-item variability. It's not unusual to calculate p-values which are less than one chance in one billion.
Personally, I prefer a simple logarithmic transformation base-10. I tell people it's easier to count zeros than to print them. The other advantage of the logarithm transformation is the ability to convert the transformed p-value into a chi-square statistic for use with inference and combining with other statistics. Thus, a value of 8 represents a probability of one chance in 100 million. I have several clients who use very small p-values as triggers for determining to investigate anomalies (I.e., one chance in one trillion or smaller).
The other reason that small p-values must be used is to prevent investigating false positives. You can use an alpha inflation correction to do this, or you can just choose a critical value which keeps the flagging rate at a manageable and practical level. The important thing is that school systems monitor and take some action so that educators are aware the data are being analyzed and they will modify their behavior. In other words, deterrence is more important than enforcement.
Dennis Maynes, Caveon Test Security
Sent from my iPad
Original Message------
The company called Caveon has done a lot of work on this. You may contact them to know more.
------------------------------
Ji Zeng
Psychometrician
Michigan Department of Education
------------------------------