Resolver Tool¶

The resolver.py tool is a manifold analysis utility that transforms abstract difficulty manifold definitions into concrete parameter grids. It provides detailed analysis of how experiment configurations will scale across different difficulty degrees and density settings.

Overview and Core Functionality¶

Resolver bridges the gap between high-level manifold specifications in task JSON files and the actual parameter combinations that will be tested. Use Resolver before running expensive evaluations to validate manifold configurations create the expected grids, compare scaling behavior across different degrees and density strategies.

The Resolver scripts implements Manifold Resolution and is used by Runner to perform:

Window Sampling: Applies degree-aware parameter selection using head/skip/body logic
Density Resampling: Reduces parameter space using corner, lowdef, or normal density strategies
Grid Analysis: Calculates total difficulty points across all manifold combinations
Configuration Validation: Ensures manifold definitions resolve correctly before execution

In standalone CLI mode, Resolver is a debug tool that parses configuration files and creates:

Summary Tables: High-level overview of difficulty points per task and density
Detailed Grids: Complete parameter value listings for each manifold configuration
Scaling Analysis: Compare computational costs across different degrees and densities

Usage¶

Basic Analysis¶

python resolver.py configs/m6.yaml 0

Analyzes all manifold tasks in the configuration at degree 0, showing: - Total difficulty points per task across normal/lowdef/corner densities - Detailed parameter grids for each manifold

Command Line Options¶

usage: resolver.py [-h] config degree

Resolve manifold definitions to concrete grids

positional arguments:
  config      Path to experiment configuration YAML file
  degree      Degree to simulate

options:
  -h, --help  show this help message and exit

Output Format¶

Summary Table¶

DIFFICULTY POINTS SUMMARY
--------------------------------------------------------------------------------
Task@Degree               Manifolds  Normal     Lowdef     Corner
--------------------------------------------------------------------------------
objects@d0                1          24         18         12
arithmetic@d0             3          26         24         16
dates@d0                  1          8          8          8
boolean@d0                1          8          6          4
movies@d0                 1          4          4          4
shuffle@d0                2          24         18         12
--------------------------------------------------------------------------------
TOTAL                                94         78         56
--------------------------------------------------------------------------------

Detailed Grid Configurations¶

Task: objects@d0
File: tasks/objects.json

  Density: normal
  ----------------------------------------
    Manifold 0: 24 points
      anchor: [0] (1 values)
      prob_adjective: [0.0] (1 values)
      distractor_count: [0, 1] (2 values)
      target_groups: [1, 2, 3] (3 values)
      length: [4, 8, 16, 24] (4 values)

Manifold Resolution Logic¶

Window Sampling¶

Resolver applies a three-stage window sampling process:

Head: Always-included values from the start of parameter ranges
Skip: Degree-aware number of values to skip after head
Body: Degree-aware number of values to include after skip

If skip+head+body exceeds range length, body values are taken from the end instead.

Density Resampling¶

After window sampling, density strategies further reduce parameter space:

Normal: No resampling (all windowed values)
Lowdef: First, middle, and last values from windowed set
Corner: First and last values only

Degree Expressions¶

Parameter counts can use degree-aware expressions:

"degree" → Current degree value
"degree+1" → Degree plus one
"max(2, degree)" → Maximum of 2 and degree

Use Cases¶

Configuration Validation¶

Manifold Debugging: Ensure parameter ranges resolve as expected
Degree Progression: Verify difficulty increases appropriately
Density Verification: Confirm resampling strategies work correctly

Resource Planning¶

Time Estimation: Predict evaluation duration based on difficulty points
Scaling Analysis: Compare computational costs across degrees

Research Planning¶

Pilot Studies: Use corner density for quick initial comparisons
Full Evaluations: Use normal density for comprehensive analysis
Publication Studies: Use lowdef density for balanced coverage

Advanced Features¶

Multi-Degree Analysis¶

Compare how manifolds scale across degrees:

for degree in 0 1 2; do
    echo "=== Degree $degree ==="
    python resolver.py configs/m6.yaml $degree | grep "TOTAL"
done

Custom Density Strategies¶

Manifold definitions can include custom resampling rules:

{
    "length": {
        "range": [8, 16, 24, 32, 40, 48],
        "window": {"skip": "degree", "body": 4},
        "resample:corner": {"first": 1, "last": 1},
        "resample:lowdef": {"first": 1, "middle": 1, "last": 1}
    }
}

Error Handling¶

Resolver provides detailed error messages for: - Invalid degree expressions - Malformed manifold definitions - Missing configuration files - Parameter range issues

Best Practices¶

Configuration Design¶

Start with corner density for rapid prototyping
Use lowdef density for balanced coverage in production
Reserve normal density for comprehensive research studies

Degree Planning¶

Degree 0: Quick model comparison and debugging
Degree 1: Standard evaluation with moderate difficulty
Degree 2+: Research-grade analysis with full complexity

Resource Management¶

Always run resolver before large evaluations
Use density resampling to fit available computational budgets
Monitor total difficulty points to estimate API costs

The resolver tool is essential for understanding and optimizing the computational requirements of ReasonScape evaluations, ensuring efficient resource usage while maintaining comprehensive difficulty coverage.