extendedmosaicperm.experiments.sign_flip module

class SignFlipExperiment(n_sims=100, nrand=100, seed=42, violation_strengths=None, sizes=None, exposures_update_interval=10)

Bases: BaseExperiment

Compare permutation-based vs sign-flip inference across generators and sizes.

The experiment runs a grid over:

• data generators: "random", "block", "common",

• size settings: e.g. "small", "medium", "large",

• residual symmetry: symmetric vs asymmetric,

• methods: permutation ("perm") vs sign-flip ("sign").

Parameters:

• n_sims (int) – Number of Monte Carlo replications per configuration.

• nrand (int) – Number of random draws (permutations or sign-flips) per fit.

• seed (int) – Base seed; seed + sim is used for the sim-th replication.

• violation_strengths (Optional[list[float]]) – List of residual correlation strengths to iterate over.

• sizes (Optional[Dict[str, Dict[str, int]]]) – Mapping from size key to a dict with keys "T", "p", "k". For example: {"small": {"T": 250, "p": 50, "k": 5}, ...}.

• exposures_update_interval (int) – If greater than zero, exposures are redrawn every given number of periods.

Examples

>>> from extendedmosaicperm.experiments.sign_flip import SignFlipExperiment
>>> exp = SignFlipExperiment(n_sims=1, nrand=3, seed=0, violation_strengths=[0.0])
>>> exp.run()  
>>> df = exp.summarize()
>>> {"label", "method", "violation_strength"}.issubset(df.columns)
True

Initialize an empty experiment container.

compare_method_timings_all_sizes(n_repeats=10, violation_strength=0.1, symmetric=True, return_full=False)

Benchmark permutation vs sign-flipping for the "block" generator.

For each size setting and method ("perm", "sign"), this routine runs n_repeats short fits using the same dataset across methods within a repeat. It records p-values and wall-clock runtimes.

Parameters:

• n_repeats (int) – Number of runs per method per size.

• violation_strength (float) – Strength of the residual correlation / symmetry violation for the "block" generator.

• symmetric (bool) – Whether residuals are symmetric (True → "sym", False → "asym").

• return_full (bool) – If True, also return the full run-level DataFrame in addition to the summary.

Returns:

If return_full is False, returns a summary DataFrame with columns:

• "size",

• "method",

• "pval_mean",

• "pval_std",

• "runtime_mean",

• "runtime_std".

If return_full is True, returns a pair (summary, df_full), where df_full contains run-level results with columns:

• "size",

• "method",

• "p_value",

• "runtime_sec".

Return type:

pandas.DataFrame or tuple[pandas.DataFrame, pandas.DataFrame]

Raises:

ValueError – If the "block" generator is not available.

extract_method(label)

Return the method identifier parsed from an experiment label.

Parameters:

label (str) – Label string of the form "<gen>_<size>_<sym|asym>_<perm|sign>".

Returns:

"sign" if the label ends with "_sign", otherwise "perm".

Return type:

str

run()

Execute the full simulation grid and store p-values.

Populates self.results as a nested dictionary of the form:

{
    label: {
        violation_strength: np.ndarray of p-values
    }
}

where label encodes generator, size, symmetry, and method.

Return type:

None