Week 6 Interactive Session: Model Evaluation & Analysis

Comprehensive evaluation and representation analysis

Course Roadmap Mapping

This week’s work in the broader GFM plan.

Week Stage Focus You will build (geogfm) Library tools Outcome
6 Stage 2: Train Foundation Model Evaluation & Analysis evaluation/visualization.py; (optional) evaluation/metrics.py matplotlib; torchmetrics optional Recon visuals; track validation loss/PSNR

Weekly goals

  • Implement reconstruction visualization utilities
  • Add a simple metric (e.g., PSNR) and validation loop hooks
  • Interpret embeddings or reconstructions qualitatively

Session Outline (and Tangled Code)

  • Concepts β†’ Components mapping
    • Reconstruction plots and side-by-sides β†’ evaluation/visualization.py
    • Simple PSNR metric (optional) β†’ evaluation/metrics.py

Package inits

# geogfm.evaluation

1) Visualization Utilities

from __future__ import annotations
import torch
import matplotlib.pyplot as plt

def show_reconstruction_grid(images: torch.Tensor, recon_patches: torch.Tensor, max_items: int = 4) -> None:
    """Show input images and reconstructed images side-by-side.
    images: (B, C, H, W)
    recon_patches: (B, N, C, P, P) reconstructed patches; will be reassembled as a naive grid.
    """
    images = images.detach().cpu()
    recon_patches = recon_patches.detach().cpu()
    b, c, h, w = images.shape
    p = recon_patches.shape[-1]
    grid_h = h // p
    grid_w = w // p

    def assemble(patches: torch.Tensor) -> torch.Tensor:
        # patches: (N, C, P, P)
        rows = []
        for r in range(grid_h):
            row = torch.cat([patches[r * grid_w + cidx] for cidx in range(grid_w)], dim=-1)
            rows.append(row)
        full = torch.cat(rows, dim=-2)
        return full

    num = min(max_items, b)
    fig, axes = plt.subplots(num, 2, figsize=(8, 4 * num))
    if num == 1:
        axes = [axes]
    for i in range(num):
        recon_full = assemble(recon_patches[i])  # (C, H, W)
        axes[i][0].imshow(images[i][0], cmap="viridis")
        axes[i][0].set_title("Input (band 1)")
        axes[i][0].axis("off")
        axes[i][1].imshow(recon_full[0], cmap="viridis")
        axes[i][1].set_title("Reconstruction (band 1)")
        axes[i][1].axis("off")
    plt.tight_layout()
    plt.show()

2) Simple Metrics (optional)

from __future__ import annotations
import torch

def psnr(pred: torch.Tensor, target: torch.Tensor, eps: float = 1e-8, max_val: float = 1.0) -> torch.Tensor:
    """Peak Signal-to-Noise Ratio over reconstructed patches.
    pred/target: (B, N, C, P, P) in [0, max_val]
    """
    mse = (pred - target) ** 2
    mse = mse.mean(dim=(-1, -2, -3, -4))  # per-sample
    return 20 * torch.log10(torch.tensor(max_val, device=pred.device)) - 10 * torch.log10(mse + eps)

Usage snippet (non-tangled)

# After obtaining a batch of images and model outputs:
# show_reconstruction_grid(images[:4], outputs["reconstructions"][:4])
# s_psnr = psnr(outputs["reconstructions"], target_patches).mean().item()
# print("PSNR:", s_psnr)