extendedmosaicperm.experiments package

Submodules

Module contents

Monte Carlo experiments and plotting utilities for extendedmosaicperm.

class AdaptiveTilingExperiment(n_sims=50, nrand=100, seed=42, violation_strengths=None, sizes=None, exposures_update_interval=10)[source]

Bases: BaseExperiment

Compare default vs adaptive tiling across generators and sizes.

For each generator/size combination, the experiment runs the test with:

  • default tiling (baseline from mosaicperm),

  • adaptive tiling (data-driven grouping, using the same test statistic).

Parameters:

  • n_sims (int) – Number of Monte Carlo replications per configuration.

  • nrand (int) – Number of random draws for the test statistic per fit.

  • seed (int) – Base seed; seed + sim is used for the sim-th replication.

  • violation_strengths (Optional[list[float]]) – List of correlation strength values to iterate over.

  • sizes (Optional[Dict[str, Dict[str, int]]]) – Mapping from size key to a dict with keys "T", "p", "k".

  • exposures_update_interval (int) – If greater than zero, exposures are redrawn every given number of periods.

Examples

>>> from extendedmosaicperm.experiments.adaptive_tiling import AdaptiveTilingExperiment
>>> exp = AdaptiveTilingExperiment(n_sims=1, nrand=3, seed=0, violation_strengths=[0.0])
>>> exp.run()  
>>> df = exp.summarize()
>>> {"label", "method", "violation_strength"}.issubset(df.columns)
True

Initialize an empty experiment container.

extract_method(label)[source]

Return the tiling mode parsed from an experiment label.

Parameters:

label (str) – Label string of the form "<gen>_<size>_<sym|asym>_<default|adaptive>".

Returns:

"adaptive" if the label ends with "_adaptive", otherwise "default".

Return type:

str

run()[source]

Execute the full simulation grid and store p-values.

Populates self.results as a nested dictionary of the form:

{
    label: {
        violation_strength: np.ndarray of p-values
    }
}

where label encodes generator, size, symmetry, and tiling mode.

Return type:

None

class BaseExperiment[source]

Bases: object

Base utilities and containers for Monte Carlo experiments.

Subclasses should implement run() to populate self.results and extract_method() to map a label string to a method identifier.

results

Nested mapping of the form {label: {violation_strength: np.ndarray of p-values}}, populated by run().

Initialize an empty experiment container.

extract_method(label)[source]

Extract the method name from a label string.

Parameters:

label (str) – Label produced by a concrete experiment, e.g. "random_small_sym_sign".

Returns:

Method identifier used by the experiment, e.g. "sign", "perm", "ridge", "ols".

Return type:

str

Raises:

NotImplementedError – If a subclass does not override this method.

flatten()[source]

Flatten nested results into a long-form DataFrame.

Returns:

A DataFrame with columns:

  • "label",

  • "violation_strength",

  • "method",

  • "p_value".

Return type:

pandas.DataFrame

Raises:

AssertionError – If self.results is None (i.e. run() has not been executed).

load(path)[source]

Load results from a pickle file.

Parameters:

path (str) – Input file path containing serialized results created by save().

Return type:

None

plot_power(df_flat, generator='random', symmetry='sym')[source]

Plot empirical power curves vs violation strength.

Creates three panels for size settings "small", "medium", and "large".

Parameters:

  • df_flat (DataFrame) – Long-form DataFrame returned by flatten().

  • generator (str) – Generator key used in labels, e.g. "random", "block", "common".

  • symmetry (str) – Symmetry flag used in labels, either "sym" or "asym".

Returns:

Figure with three subplots showing power by method.

Return type:

matplotlib.figure.Figure

plot_pval_histogram(df_flat, label, v=0.0, bins=20)[source]

Plot histograms of p-values at a given violation strength.

Parameters:

  • df_flat (DataFrame) – Long-form DataFrame returned by flatten().

  • label (str) – Label prefix to filter rows, e.g. "random_small_sym".

  • v (float) – Violation strength value to slice on.

  • bins (int) – Number of histogram bins.

Returns:

Figure with two panels, one histogram per method.

Return type:

matplotlib.figure.Figure

plot_qq(df_flat, label_bases, v)[source]

Draw QQ-plots of p-values for selected label bases at a fixed violation level.

Parameters:

  • df_flat (DataFrame) – Long-form DataFrame returned by flatten().

  • label_bases (list[str]) – Base labels (without method suffix) to compare, e.g. ["random_small_sym", "random_medium_sym"].

  • v (float) – Violation strength value to slice on.

Returns:

Figure with two QQ plots (empirical vs uniform).

Return type:

matplotlib.figure.Figure

save(path)[source]

Serialize results to a pickle file.

Parameters:

path (str) – Output file path for the serialized results.

Return type:

None

summarize()[source]

Summarize empirical power and p-value moments.

For each label and violation strength, compute empirical rejection probability and basic p-value summaries.

Returns:

A summary DataFrame with columns:

  • "label",

  • "method",

  • "violation_strength",

  • "power",

  • "mean_pval",

  • "median_pval",

  • "std_pval",

  • "n".

Return type:

pandas.DataFrame

Raises:

AssertionError – If self.results is None.

class RidgeExperiment(n_sims=100, nrand=100, seed=42, violation_strengths=None, sizes=None, exposures_update_interval=10)[source]

Bases: BaseExperiment

Compare OLS vs RidgeCV residual estimation across generators and sizes.

The experiment runs a grid over:

  • generators: "random", "block", "common",

  • sizes: e.g. "small", "medium", "large",

  • residual symmetry: symmetric vs asymmetric,

  • residual estimators: OLS vs RidgeCV (with a grid of alphas).

Parameters:

  • n_sims (int) – Number of Monte Carlo replications per configuration.

  • nrand (int) – Number of random draws for the test statistic per fit.

  • seed (int) – Base seed; seed + sim is used for the sim-th replication.

  • violation_strengths (Optional[list[float]]) – List of correlation strength values to iterate over.

  • sizes (Optional[Dict[str, Dict[str, int]]]) – Mapping from size key to a dict with keys "T", "p", "k".

  • exposures_update_interval (int) – If greater than zero, exposures are redrawn every given number of periods.

Examples

>>> from extendedmosaicperm.experiments.ridge import RidgeExperiment
>>> exp = RidgeExperiment(n_sims=1, nrand=3, seed=0, violation_strengths=[0.0])
>>> exp.run()  
>>> df = exp.summarize()
>>> {"label", "method", "violation_strength"}.issubset(df.columns)
True

Initialize an empty experiment container.

extract_method(label)[source]

Return the residual estimator identifier parsed from a label.

Parameters:

label (str) – Label string of the form "<gen>_<size>_<sym|asym>_<ols|ridge>".

Returns:

"ridge" if the label ends with "_ridge", otherwise "ols".

Return type:

str

run()[source]

Execute the full simulation grid and store p-values.

Populates self.results as a nested dictionary of the form:

{
    label: {
        violation_strength: np.ndarray of p-values
    }
}

where label encodes generator, size, symmetry, and residual method.

Return type:

None

class SignFlipExperiment(n_sims=100, nrand=100, seed=42, violation_strengths=None, sizes=None, exposures_update_interval=10)[source]

Bases: BaseExperiment

Compare permutation-based vs sign-flip inference across generators and sizes.

The experiment runs a grid over:

  • data generators: "random", "block", "common",

  • size settings: e.g. "small", "medium", "large",

  • residual symmetry: symmetric vs asymmetric,

  • methods: permutation ("perm") vs sign-flip ("sign").

Parameters:

  • n_sims (int) – Number of Monte Carlo replications per configuration.

  • nrand (int) – Number of random draws (permutations or sign-flips) per fit.

  • seed (int) – Base seed; seed + sim is used for the sim-th replication.

  • violation_strengths (Optional[list[float]]) – List of residual correlation strengths to iterate over.

  • sizes (Optional[Dict[str, Dict[str, int]]]) – Mapping from size key to a dict with keys "T", "p", "k". For example: {"small": {"T": 250, "p": 50, "k": 5}, ...}.

  • exposures_update_interval (int) – If greater than zero, exposures are redrawn every given number of periods.

Examples

>>> from extendedmosaicperm.experiments.sign_flip import SignFlipExperiment
>>> exp = SignFlipExperiment(n_sims=1, nrand=3, seed=0, violation_strengths=[0.0])
>>> exp.run()  
>>> df = exp.summarize()
>>> {"label", "method", "violation_strength"}.issubset(df.columns)
True

Initialize an empty experiment container.

compare_method_timings_all_sizes(n_repeats=10, violation_strength=0.1, symmetric=True, return_full=False)[source]

Benchmark permutation vs sign-flipping for the "block" generator.

For each size setting and method ("perm", "sign"), this routine runs n_repeats short fits using the same dataset across methods within a repeat. It records p-values and wall-clock runtimes.

Parameters:

  • n_repeats (int) – Number of runs per method per size.

  • violation_strength (float) – Strength of the residual correlation / symmetry violation for the "block" generator.

  • symmetric (bool) – Whether residuals are symmetric (True → "sym", False → "asym").

  • return_full (bool) – If True, also return the full run-level DataFrame in addition to the summary.

Returns:

If return_full is False, returns a summary DataFrame with columns:

  • "size",

  • "method",

  • "pval_mean",

  • "pval_std",

  • "runtime_mean",

  • "runtime_std".

If return_full is True, returns a pair (summary, df_full), where df_full contains run-level results with columns:

  • "size",

  • "method",

  • "p_value",

  • "runtime_sec".

Return type:

pandas.DataFrame or tuple[pandas.DataFrame, pandas.DataFrame]

Raises:

ValueError – If the "block" generator is not available.

extract_method(label)[source]

Return the method identifier parsed from an experiment label.

Parameters:

label (str) – Label string of the form "<gen>_<size>_<sym|asym>_<perm|sign>".

Returns:

"sign" if the label ends with "_sign", otherwise "perm".

Return type:

str

run()[source]

Execute the full simulation grid and store p-values.

Populates self.results as a nested dictionary of the form:

{
    label: {
        violation_strength: np.ndarray of p-values
    }
}

where label encodes generator, size, symmetry, and method.

Return type:

None

compute_power_table(df_flat, alpha=0.05)[source]

Aggregate empirical power across methods and configurations.

Parameters:

  • df_flat (DataFrame) – Long-form DataFrame from BaseExperiment.flatten().

  • alpha (float) – Significance level used to compute power (default 0.05).

Returns:

DataFrame with columns:

  • "gen",

  • "size",

  • "sym",

  • "method",

  • "violation_strength",

  • "power".

Return type:

pandas.DataFrame

enrich_flatten(df_flat)[source]

Add parsed label components as columns to a flattened DataFrame.

Parameters:

df_flat (DataFrame) – Long-form DataFrame from BaseExperiment.flatten(), containing at least a "label" column.

Returns:

Copy of df_flat with extra columns added:

  • "gen",

  • "size",

  • "sym",

  • "method".

Return type:

pandas.DataFrame

plot_figure3_panels(L_time, simulate_fn, *, test_stat=<function mean_maxcorr_stat>, n_rep_a=100, n_rep_b=100, b_boot=300, n_runs_c=100, n_perm_c=300, bins_a=25, bins_b=25, bins_c=25, seed=42, colors=None, outdir=None, save_basename='figure3_panels', title_fontsize=14, label_fontsize=12, tick_fontsize=11, legend_fontsize=11)[source]

Reproduce a three-panel comparison: naive permutation, bootstrap, and Mosaic test.

Panel (a) compares the naive permutation distribution of the maximum off-diagonal correlation with the observed OLS-based statistic. Panel (b) compares bootstrap Z-statistics with a standard normal. Panel (c) compares Mosaic test statistics with their permutation null.

Parameters:

  • L_time (ndarray) – Time-varying exposures of shape (T, p, k).

  • simulate_fn (Callable[[Generator], ndarray]) – Callable that takes a NumPy Generator and returns a simulated outcome matrix Y of shape (T, p).

  • test_stat (Callable) – Test statistic to be used by the Mosaic test.

  • n_rep_a (int) – Number of datasets for panel (a).

  • n_rep_b (int) – Number of datasets for panel (b).

  • b_boot (int) – Number of bootstrap draws per dataset in panel (b).

  • n_runs_c (int) – Number of datasets for panel (c).

  • n_perm_c (int) – Number of permutations per dataset in panel (c).

  • bins_a (int) – Number of bins for the histogram in panel (a).

  • bins_b (int) – Number of bins for the histogram in panel (b).

  • bins_c (int) – Number of bins for the histogram in panel (c).

  • seed (int) – Seed used to initialize the random generator.

  • colors (Optional[Dict[str, str]]) – Optional mapping for color names {"null": ..., "alt": ..., "boot": ...}.

  • outdir (Optional[str]) – Optional output directory to save the figure as PNG.

  • save_basename (str) – Base file name for saving (without extension).

  • title_fontsize (int) – Font size for panel titles.

  • label_fontsize (int) – Font size for axis labels.

  • tick_fontsize (int) – Font size for tick labels.

  • legend_fontsize (int) – Font size for legend entries.

Returns:

The figure and a dictionary with the simulated statistics.

Return type:

tuple[matplotlib.figure.Figure, Dict[str, np.ndarray]]

plot_qq_grid_all_sizes(df_flat, generators=None, sizes=None, v=0.0, symmetry_options=('sym', 'asym'), method_linestyles=None, figscale=(12, 16), output_path=None, title_fontsize=17, label_fontsize=16, tick_fontsize=15, legend_fontsize=14, legend_ncol=None, legend_offset=-0.012, tight_rect=(0.02, 0.05, 0.98, 0.96))[source]

Create a QQ-grid by generator (rows) and symmetry (columns).

Each panel shows QQ-plots of p-values at a fixed violation level v, colored by size and styled by method.

Parameters:

  • df_flat (DataFrame) – Long-form DataFrame from BaseExperiment.flatten().

  • generators (Optional[Iterable[str]]) – Generators to display. If None, all available generators are used.

  • sizes (Optional[Iterable[str]]) – Size categories to display. If None, all available sizes are used.

  • v (float) – Violation strength to slice on.

  • symmetry_options (Iterable[str]) – Iterable of symmetry flags (e.g. ("sym", "asym")).

  • method_linestyles (Optional[Dict[str, str]]) – Mapping from method to line style, e.g. {"perm": "-", "sign": "--"}. If None, a default mapping is used.

  • figscale (Tuple[int, int]) – Figure size as (width, height) in inches.

  • output_path (Optional[str]) – Optional path to save the figure as an image.

  • title_fontsize (int) – Font size for panel titles.

  • label_fontsize (int) – Font size for axis labels.

  • tick_fontsize (int) – Font size for tick labels.

  • legend_fontsize (int) – Font size for legend labels.

  • legend_ncol (Optional[int]) – Number of columns in the combined legend. If None, a heuristic is used.

  • legend_offset (float) – Vertical offset for the combined legend in figure coordinates.

  • tight_rect (Tuple[float, float, float, float]) – Bounding rectangle for matplotlib.pyplot.tight_layout().

Returns:

The created figure.

Return type:

matplotlib.figure.Figure

Raises:

ValueError – If the required label structure or requested generators / sizes are not present in the data.

plot_qq_grid_generators_by_alpha(df_flat, generators, alphas, sizes=None, symmetry='asym', method_linestyles=None, figscale_base=(5.8, 4.8), output_path=None, title_fontsize=17, label_fontsize=16, tick_fontsize=15, legend_fontsize=14, legend_ncol=3, legend_offset=-0.012, tight_rect=(0.02, 0.05, 0.98, 0.94))[source]

Create a QQ-grid with rows = generators and columns = violation strengths.

Parameters:

  • df_flat (DataFrame) – Long-form DataFrame from BaseExperiment.flatten().

  • generators (Iterable[str]) – Generators to include as rows.

  • alphas (Iterable[float]) – Violation strength values (x-axis conditions) to include as columns.

  • sizes (Optional[Iterable[str]]) – Size categories to display. If None, all available sizes are used.

  • symmetry (str) – Symmetry flag to filter on (e.g. "asym").

  • method_linestyles (Optional[Dict[str, str]]) – Mapping from method to line style. If None, a default mapping is used.

  • figscale_base (Tuple[float, float]) – Base figure size for a single panel. Total figure size scales with the grid dimensions.

  • output_path (Optional[str]) – Optional path to save the figure as an image.

  • title_fontsize (int) – Font size for panel titles.

  • label_fontsize (int) – Font size for axis labels.

  • tick_fontsize (int) – Font size for tick labels.

  • legend_fontsize (int) – Font size for legend labels.

  • legend_ncol (int) – Number of columns in the combined legend.

  • legend_offset (float) – Vertical offset for the combined legend.

  • tight_rect (Tuple[float, float, float, float]) – Bounding rectangle for matplotlib.pyplot.tight_layout().

Returns:

The created figure.

Return type:

matplotlib.figure.Figure

Raises:

ValueError – If there is no data after filtering or requested sizes are not present.