Psychometric Data Generator - User Guide

The **Psychometric Data Generator** is a powerful tool designed to create realistic test datasets with valid psychometric metrics for WASPL assessments. This tool generates simulated student responses that maintain statistically sound characteristics, making it ideal for testing, demonstrations, training, and quality validation.

Purpose and Applications

### Primary Uses

- **Testing & Validation**: Generate datasets to test WASPL's analytical capabilities

- **Demonstrations**: Create realistic data for showcasing platform features

- **Training**: Provide educational datasets for learning psychometric concepts

- **Quality Assurance**: Test detection algorithms with known data characteristics

- **Research**: Generate controlled datasets for psychometric research

### Key Benefits

- **Realistic Data**: Simulated responses follow actual response patterns

- **Controlled Quality**: Target specific reliability coefficients (Cronbach's α)

- **Instant Generation**: Create datasets in seconds rather than months

- **Educational Value**: Understand the relationship between item quality and test reliability

## What the Generator Creates

The Psychometric Data Generator produces:

### 1. Student Response Data

- **Individual Responses**: Simulated answers for each student to each test item

- **Response Patterns**: Realistic distribution following Item Response Theory (IRT)

- **Consistency Modeling**: Variable response consistency based on student ability

### 2. Psychometric Metrics

- **Cronbach's Alpha**: Test reliability coefficient (internal consistency)

- **Item Discrimination**: How well items differentiate between students

- **Item Difficulty**: Distribution of item difficulty parameters

- **Response Timing**: Realistic completion times per item

### 3. Statistical Properties

- **Score Distribution**: Normal or custom distributions of total scores

- **Item-Total Correlations**: Relationships between item and total performance

- **Standard Errors**: Measurement precision indicators

- **Missing Data**: Realistic patterns of incomplete responses

## Quick Start Presets

The generator offers three pre-configured presets for immediate use:

### 🎯 Realistic Demo

- **Target**: α ≥ 0.85 (Grade B)

- **Quality**: High-quality items (80% good items)

- **Use Case**: Professional demonstrations and standard testing

- **Characteristics**: Balanced difficulty, good discrimination

### 🔍 Detection Test

- **Target**: α ≈ 0.40 (Grade D)

- **Quality**: Mixed quality with problematic items

- **Use Case**: Testing quality detection algorithms

- **Characteristics**: Includes poor items, low reliability

### 📚 Educational Training

- **Target**: α ≥ 0.75 (Grade C)

- **Quality**: Acceptable quality for learning

- **Use Case**: Training and educational purposes

- **Characteristics**: Moderate quality, instructional value

## Expert Mode Configuration

For advanced users, Expert Mode provides full control over generation parameters:

### Core Parameters

- **Target Cronbach's Alpha**: Set desired reliability (0.5 - 0.95)

- **Minimum Discrimination**: Item quality threshold (0.1 - 0.6)

- **Response Consistency**: Student behavior variability (0.1 - 0.8)

- **Sample Size**: Number of students to simulate

- **Missing Data Rate**: Percentage of incomplete responses

### Advanced Options

- **Timing Generation**: Include realistic completion times

- **Debug Mode**: Additional diagnostic information

- **Custom Distributions**: Specify ability and difficulty distributions

## Cronbach's Alpha Categories (A, B, C, D)

The generator uses standard psychometric thresholds to categorize test reliability:

### Category A - Excellent (α ≥ 0.9)

- **Interpretation**: Outstanding reliability

- **Suitable For**: High-stakes testing, certification exams

- **Characteristics**: Very consistent measurement, minimal measurement error

- **Color Code**: Green (#28a745)

### Category B - Good (0.8 ≤ α < 0.9)

- **Interpretation**: Good reliability

- **Suitable For**: Most educational assessments, research

- **Characteristics**: Reliable measurement with acceptable error

- **Color Code**: Teal (#17a2b8)

### Category C - Acceptable (0.7 ≤ α < 0.8)

- **Interpretation**: Acceptable reliability

- **Suitable For**: Formative assessment, initial testing

- **Characteristics**: Adequate for most purposes, some measurement error

- **Color Code**: Yellow (#ffc107)

### Category D - Insufficient (α < 0.7)

- **Interpretation**: Poor reliability

- **Suitable For**: Pilot testing, diagnostic purposes only

- **Characteristics**: High measurement error, results should be interpreted cautiously

- **Color Code**: Red (#dc3545)

## Generation Process

### Step 1: Configuration

1. Select a Quick Start preset or choose Expert Mode

2. Configure generation parameters

3. Select target test and publication(s)

4. Review settings and estimated generation time

### Step 2: Validation

- System validates configuration parameters

- Checks for realistic parameter combinations

- Estimates generation time and resource requirements

### Step 3: Generation

- Creates simulated response matrix

- Applies psychometric models (IRT/CTT)

- Calculates reliability and item statistics

- Generates timing data (if enabled)

### Step 4: Results

- Displays generation summary

- Shows achieved vs. target metrics

- Provides data quality indicators

- Saves results to selected publication(s)

## Technical Specifications

### Supported Models

- **Classical Test Theory (CTT)**: Traditional reliability analysis

- **Item Response Theory (IRT)**: Modern psychometric modeling

- **Rasch Model**: Specific IRT implementation for dichotomous items

### Data Format

- **Response Matrix**: Students × Items binary/polytomous responses

- **Metadata**: Student IDs, item parameters, session information

- **Timing Data**: Response times in milliseconds

- **Quality Metrics**: Comprehensive psychometric statistics

### Performance

- **Small Datasets** (< 50 students): < 1 second

- **Medium Datasets** (50-200 students): 1-2 seconds

- **Large Datasets** (200+ students): 2-5 seconds

## Best Practices

### For Demonstrations

- Use "Realistic Demo" preset

- Target α ≥ 0.85 for professional appearance

- Include timing data for realistic simulation

### For Testing & QA

- Use "Detection Test" preset for algorithm validation

- Mix high and low quality items

- Test edge cases with extreme parameters

### For Training

- Use "Educational Training" preset

- Show progression from poor to excellent reliability

- Demonstrate impact of item quality on overall test reliability

### For Research

- Use Expert Mode for precise control

- Document all parameter settings

- Validate against real data when possible

## Troubleshooting

### Common Issues

- **Generation Fails**: Check parameter ranges and test selection

- **Poor Quality Results**: Adjust discrimination thresholds

- **Unrealistic Data**: Review consistency and timing parameters

### Performance Optimization

- Limit student count for faster generation

- Disable timing data if not needed

- Use appropriate quality thresholds

## Integration with WASPL

The generated data integrates seamlessly with:

- **Results Analysis**: Full psychometric reporting

- **CAT System**: Adaptive testing calibration

- **Quality Dashboard**: Real-time monitoring

- **Export Functions**: Multiple format support

---

*This tool is part of the WASPL Developer Tools suite, designed to support comprehensive assessment development and validation workflows.*