edu.umass.cs.surveyman.qc

Class QCMetrics

java.lang.Object

edu.umass.cs.surveyman.qc.QCMetrics

public class QCMetrics
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static int	bootstrapIterations
static MersenneRandom	rng

Constructor Summary

Constructors

Constructor and Description
QCMetrics(Survey survey)
QCMetrics(Survey survey, boolean smoothing)

Method Summary

Methods

Modifier and Type	Method and Description
double	averagePathLength() Simulates the survey for 5000 uniform random respondents and returns the average path length.
BreakoffByPosition	calculateBreakoffByPosition(List<? extends SurveyResponse> responses) Aggregates the breakoff according to the last position answered.
BreakoffByQuestion	calculateBreakoffByQuestion(List<? extends SurveyResponse> responses) Aggregates the breakoff according to which question was last answered.
WordingBiasStruct	calculateWordingBiases(List<? extends SurveyResponse> responses, double alpha) Searches for significant wording biases observed in survey responses.
protected static double	cellExpectation(int[][] contingencyTable, int i, int j, int n)
static double	chiSquared(int[][] contingencyTable, Object[] categoryA, Object[] categoryB) Returns the chi-squared statistic for the input data.
static double	chiSquareTest(int df, double testStatistic)
ClassifiedRespondentsStruct	classifyResponses(List<? extends SurveyResponse> responses, Classifier classifier, double alpha) Classifies the input responses according to the classifier.
protected List<Double>	computeMeans(SurveyResponse sr, List<? extends SurveyResponse> responses, Classifier classifier)
protected static void	computeRanks(double[] xranks, List xs)
protected static double	cramersV(Map<String,IQuestionResponse> listA, Map<String,IQuestionResponse> listB)
boolean	entropyClassification(SurveyResponse sr, List<? extends SurveyResponse> responses, double alpha) Return true if the response is valid, on the basis of an entropy-based metric.
static List<Question>	filterAnalyzable(List<Question> questionList) Filter the list of questions for things that can be analyzed in terms of survey correctness.
static List<List<SurveyResponse>>	generateBootstrapSample(List<? extends SurveyResponse> responses) Generates the bootstrap sample for the input response and the specified number of iterations.
double	getEntropyForResponse(SurveyResponse surveyResponse)
protected static List<SurveyDatum>	getEquivalentAnswerVariants(Question q, SurveyDatum c) Returns equivalent answer options (a list of survey.SurveyDatum)
Map<Question,Map<Question,CorrelationStruct>>	getFrequenciesOfRandomCorrelation() Simulates a survey of 100% random uniform respondents over sampleSize and calculates a prior on false correlation.
double	getLLForResponse(SurveyResponse surveyResponse)
double	getMaxPossibleEntropy() The public method used to compute the maximum number of bits needed to represent this survey.
static boolean	isAnalyzable(Question question)
boolean	isFinalQuestion(Question question, SurveyResponse surveyResponse)
boolean	logLikelihoodClassification(SurveyResponse sr, List<? extends SurveyResponse> responses, double alpha) Returns true if the response is valid, on the basis of the log likelihood.
void	lpoClassification(List<? extends SurveyResponse> responses, double epsilon) Computes the validity of the input responses, based on the "Least popular option" metric.
void	makeFrequencies(List<? extends SurveyResponse> responses) Creates a frequency map for the actual responses to the survey.
void	makeProbabilities(List<? extends SurveyResponse> responses)
static double	mannWhitney(Question q1, Question q2, List<SurveyDatum> list1, List<SurveyDatum> list2) Mann-Whitney statistic, specialized for comparing survey questions.
int	maximumPathLength()
int	minimumPathLength()
protected static double	spearmansRho(Map<String,IQuestionResponse> listA, Map<String,IQuestionResponse> listB)
static double	surveyEntropy(Survey survey, List<? extends SurveyResponse> responses) Calculates the empirical entropy for this survey, given a set of responses.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

rng

public static final MersenneRandom rng

bootstrapIterations

public static int bootstrapIterations

Constructor Detail

QCMetrics

public QCMetrics(Survey survey)

QCMetrics

public QCMetrics(Survey survey,
         boolean smoothing)

Method Detail

maximumPathLength

public int maximumPathLength()

isAnalyzable

public static boolean isAnalyzable(Question question)

filterAnalyzable

public static List<Question> filterAnalyzable(List<Question> questionList)

Filter the list of questions for things that can be analyzed in terms of survey correctness. This means questions that are not instructional, not custom questions, and not freetext.

Parameters:

questionList - The list to filter.

Returns:

A new list consisting only of analyzable questions.

getEquivalentAnswerVariants

protected static List<SurveyDatum> getEquivalentAnswerVariants(Question q,
                                            SurveyDatum c)

Returns equivalent answer options (a list of survey.SurveyDatum)

Parameters:

q - The question whose variants we want. If there are no variants, then a set of just this question is returned.

c - The answer the respondent provided for this question.

Returns:

A list of the equivalent answers.

surveyEntropy

public static double surveyEntropy(Survey survey,
                   List<? extends SurveyResponse> responses)

Calculates the empirical entropy for this survey, given a set of responses.

Parameters:

survey - The survey these respondents answered.

responses - The list of actual or simulated responses to the survey.

Returns:

The caluclated base-2 entropy.

getMaxPossibleEntropy

public double getMaxPossibleEntropy()

The public method used to compute the maximum number of bits needed to represent this survey.

Returns:

The maximum possible entropy for this source.

minimumPathLength

public int minimumPathLength()

averagePathLength

public double averagePathLength()
                         throws SurveyException

Simulates the survey for 5000 uniform random respondents and returns the average path length.

Returns:

average path length.

Throws:

SurveyException

makeFrequencies

public void makeFrequencies(List<? extends SurveyResponse> responses)

Creates a frequency map for the actual responses to the survey.

Parameters:

responses - The list of actual or simulated responses to the survey.

makeProbabilities

public void makeProbabilities(List<? extends SurveyResponse> responses)

getLLForResponse

public double getLLForResponse(SurveyResponse surveyResponse)
                        throws SurveyException

Throws:

SurveyException

getEntropyForResponse

public double getEntropyForResponse(SurveyResponse surveyResponse)
                             throws SurveyException

Throws:

SurveyException

generateBootstrapSample

public static List<List<SurveyResponse>> generateBootstrapSample(List<? extends SurveyResponse> responses)

Generates the bootstrap sample for the input response and the specified number of iterations. Default 2000.

Parameters:

responses - The list of actual or simulated responses to the survey.

Returns:

The bootstrapped sample of possible survey responses.

computeMeans

protected List<Double> computeMeans(SurveyResponse sr,
                        List<? extends SurveyResponse> responses,
                        Classifier classifier)
                             throws SurveyException

Throws:

SurveyException

logLikelihoodClassification

public boolean logLikelihoodClassification(SurveyResponse sr,
                                  List<? extends SurveyResponse> responses,
                                  double alpha)
                                    throws SurveyException

Returns true if the response is valid, on the basis of the log likelihood.

Parameters:

sr - The survey response we are classifying.

responses - The list of actual or simulated responses to the survey

alpha - The cutoff used for determining whether a likelihood is too low (a percentage of area under the curve).

Returns:

Decision indicating whether the input response is valid.

Throws:

SurveyException

lpoClassification

public void lpoClassification(List<? extends SurveyResponse> responses,
                     double epsilon)
                       throws SurveyException

Computes the validity of the input responses, based on the "Least popular option" metric.

Parameters:

responses -

epsilon -

Throws:

SurveyException

entropyClassification

public boolean entropyClassification(SurveyResponse sr,
                            List<? extends SurveyResponse> responses,
                            double alpha)
                              throws SurveyException

Return true if the response is valid, on the basis of an entropy-based metric.

Parameters:

sr - The survey response we are classifying.

responses - The list of actual or simulated responses to the survey

alpha - The cutoff used for determining whether a likelihood is too low (a percentage of area under the curve).

Returns:

boolean indicating whether the response is valid.

Throws:

SurveyException

computeRanks

protected static void computeRanks(double[] xranks,
                List xs)

spearmansRho

protected static double spearmansRho(Map<String,IQuestionResponse> listA,
                  Map<String,IQuestionResponse> listB)

cellExpectation

protected static double cellExpectation(int[][] contingencyTable,
                     int i,
                     int j,
                     int n)

chiSquared

public static double chiSquared(int[][] contingencyTable,
                Object[] categoryA,
                Object[] categoryB)

Returns the chi-squared statistic for the input data.

Parameters:

contingencyTable -

categoryA -

categoryB -

Returns:

chiSquareTest

public static double chiSquareTest(int df,
                   double testStatistic)

cramersV

protected static double cramersV(Map<String,IQuestionResponse> listA,
              Map<String,IQuestionResponse> listB)

mannWhitney

public static double mannWhitney(Question q1,
                 Question q2,
                 List<SurveyDatum> list1,
                 List<SurveyDatum> list2)

Mann-Whitney statistic, specialized for comparing survey questions.

Parameters:

q1 -

q2 -

list1 -

list2 -

Returns:

getFrequenciesOfRandomCorrelation

public Map<Question,Map<Question,CorrelationStruct>> getFrequenciesOfRandomCorrelation()
                                                                                throws SurveyException

Simulates a survey of 100% random uniform respondents over sampleSize and calculates a prior on false correlation.

Returns:

Empirical false correlation.

Throws:

SurveyException

isFinalQuestion

public boolean isFinalQuestion(Question question,
                      SurveyResponse surveyResponse)

calculateBreakoffByPosition

public BreakoffByPosition calculateBreakoffByPosition(List<? extends SurveyResponse> responses)

Aggregates the breakoff according to the last position answered.

Parameters:

responses - The list of actual or simulated responses to the survey.

Returns:

A BreakoffByPosition object containing all of the values just computed.

calculateBreakoffByQuestion

public BreakoffByQuestion calculateBreakoffByQuestion(List<? extends SurveyResponse> responses)

Aggregates the breakoff according to which question was last answered.

Parameters:

responses - The list of actual or simulated responses to the survey.

Returns:

A BreakoffByQuestion object containing all of the values just computed.

calculateWordingBiases

public WordingBiasStruct calculateWordingBiases(List<? extends SurveyResponse> responses,
                                       double alpha)
                                         throws SurveyException

Searches for significant wording biases observed in survey responses.

Parameters:

responses - The list of actual or simulated responses to the survey.

alpha - The cutoff used for determining whether the bias is significant.

Returns:

A WordingBiasStruct object containing all of the values just computed.

Throws:

SurveyException

classifyResponses

public ClassifiedRespondentsStruct classifyResponses(List<? extends SurveyResponse> responses,
                                            Classifier classifier,
                                            double alpha)
                                              throws SurveyException

Classifies the input responses according to the classifier. The DynamicSurveyResponse objects will hold the computed classification, and the method will return a classification structure for easy printing and jsonizing.

Parameters:

responses - The list of actual or simulated responses to the survey.

classifier - The enum corresponding to the classifier type.

alpha - The cutoff used for determining whether validity is exceptionally low.

Returns:

A ClassifiedRespondentsStruct object containing all of the values just computed.

Throws:

SurveyException