This notebook uses k-means to group a photo's pixels, then applies an object detector to regions within a cluster.
The notebook covers three methods:
- K-means: assign pixels to clusters and update the cluster centers.
- KNN: nearby labeled examples vote.
- Regression: use a trend to predict a number.
Each section introduces a different part of the algorithm. Use the sample image first, then compare the results after changing a setting.
1 · Get ready
Do: run the next cell once. It loads the tools; you can open its code later.
In Colab: save your own copy, choose a CPU runtime, and run cells from the top with ▶.
Setup and privacy notes
A Google account is needed to run Colab. No GPU or API key is needed. Downloads require internet. Your uploaded image goes to your Colab runtime, not this website. Outputs can contain your photo; check before sharing. After a runtime restart, rerun from the top.Start the toolsOpen the Python code
#@title Start the tools
import importlib.util
import subprocess
import sys
packages = {"numpy": "numpy>=1.26,<3", "PIL": "pillow>=10,<13",
"matplotlib": "matplotlib>=3.8,<4", "scipy": "scipy>=1.11,<2",
"ipywidgets": "ipywidgets>=8,<9"}
missing = [spec for name, spec in packages.items() if importlib.util.find_spec(name) is None]
if missing:
subprocess.check_call([sys.executable, "-m", "pip", "install", "-q", *missing])
import io
import json
import hashlib
import urllib.request
from pathlib import Path
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
from matplotlib.patches import Rectangle
from PIL import Image, ImageOps
from scipy import ndimage
from IPython.display import display
plt.rcParams.update({"figure.dpi": 110, "axes.titlesize": 11})
print("Ready. You will write the clustering functions below.")
2 · Start with the dog
Predict: will the white fur and green grass land in different groups?
Do: run with USE_UPLOAD = False. Save your own photo for the remix round.
Notice: you can name the dog. Our clustering algorithm only receives pixel measurements.
Using your own image
Set USE_UPLOAD to True and choose one image, then rerun all cells below. JPEG, PNG and WebP work; convert HEIC/RAW first. Use a file under 20 MB. Images keep their shape, transparency goes on white, and animated files use their first frame. Sample source: PyTorch Hub.
USE_UPLOAD = False # Change to True to choose your own image.
SAMPLE_URL = "https://eloc-cyber.com/lab/clustering/assets/dog.jpg"
if USE_UPLOAD:
from google.colab import files
uploaded = files.upload()
if len(uploaded) != 1:
raise ValueError("Choose exactly one image, then rerun this cell.")
image_name, image_bytes = next(iter(uploaded.items()))
else:
image_name = "sample-dog.jpg"
with urllib.request.urlopen(SAMPLE_URL, timeout=30) as response:
image_bytes = response.read()
if len(image_bytes) > 20 * 1024 * 1024:
raise ValueError("Use an image smaller than 20 MB. Export a smaller JPEG or PNG and try again.")
def prepare_image(data, max_side=640):
"""Read the first frame, correct orientation, and preserve the whole image."""
with Image.open(io.BytesIO(data)) as opened:
opened.seek(0)
rgba = ImageOps.exif_transpose(opened).convert("RGBA")
background = Image.new("RGBA", rgba.size, "white")
rgb = Image.alpha_composite(background, rgba).convert("RGB")
rgb.thumbnail((max_side, max_side), Image.Resampling.LANCZOS)
return rgb
source_image = prepare_image(image_bytes)
image_sha256 = hashlib.sha256(image_bytes).hexdigest()
display(source_image)
print(image_name, "→ working image:", source_image.size)
3 · Tell the algorithm what matters
One idea: a pixel becomes a row of numbers: red, green, blue, and position.
Do: start with K = 5. That asks for five groups.
Predict: if position stops mattering, could white pixels far apart join the same group?
POSITION_WEIGHT = 0 means color only. A larger value makes location matter more. Keep the other settings for now.
K = 5
POSITION_WEIGHT = 0.3
SEED = 7
MAX_ITERATIONS = 20
CLUSTER_SIDE = 192
if not isinstance(K, int) or not 2 <= K <= 10:
raise ValueError("Choose an integer K from 2 to 10.")
if not 0 <= POSITION_WEIGHT <= 3 or not 1 <= MAX_ITERATIONS <= 40:
raise ValueError("Use a position weight from 0 to 3 and 1–40 iterations.")
small_image = source_image.copy()
small_image.thumbnail((CLUSTER_SIDE, CLUSTER_SIDE), Image.Resampling.LANCZOS)
rgb = np.asarray(small_image, dtype=np.float32) / 255.0
height, width = rgb.shape[:2]
y_grid, x_grid = np.mgrid[0:height, 0:width]
positions = np.column_stack((x_grid.ravel() / max(width - 1, 1),
y_grid.ravel() / max(height - 1, 1)))
X = np.column_stack((rgb.reshape(-1, 3), positions * POSITION_WEIGHT)).astype(np.float32)
if K > len(X):
raise ValueError("This picture has fewer pixels than groups. Use a larger image or a smaller K.")
palette = np.array([[0.94,0.55,0.28],[0.24,0.64,0.81],[0.66,0.46,0.76],
[0.88,0.76,0.24],[0.29,0.69,0.52],[0.91,0.38,0.49],
[0.44,0.49,0.80],[0.63,0.70,0.32],[0.72,0.48,0.30],
[0.38,0.73,0.72]])
print(f"{width} × {height} = {len(X):,} pixels; each row is [red, green, blue, weighted x, weighted y].")
print("First five rows:\n", np.round(X[:5], 3))
Look at the pixels as dots
Do: run the plot. Each dot is a real pixel, drawn in its original color.
Notice: the axes show red and green. Our algorithm also sees blue and position.
Say it: “Similar means similar in the measurements we choose.”
Look at pixels as dotsOpen the Python code
#@title Look at pixels as dots
preview_ids = np.random.default_rng(SEED).choice(len(X), min(1500, len(X)), replace=False)
fig, ax = plt.subplots(figsize=(5, 4))
ax.scatter(X[preview_ids, 0], X[preview_ids, 1], c=rgb.reshape(-1,3)[preview_ids], s=7)
ax.set(xlabel="Red (0–1)", ylabel="Green (0–1)", title="Pixels as measurements")
plt.show()
4 · Find the nearest center
One idea: smaller distance means a closer match.
Predict: will [0.9, 0.1, 0.1] join the red center or the blue one?
Do: run the example. Then make the test pixel more blue and rerun these two small cells.
Notice: no dog, grass, or sky labels appear in the calculation.
def squared_distances(points, centers):
# Shape: pixels × centers × measurements. Sum across measurements.
differences = points[:, None, :] - centers[None, :, :]
return (differences ** 2).sum(axis=2)
example_pixel = np.array([[0.9, 0.1, 0.1]])
example_centers = np.array([[1.0, 0.0, 0.0], [0.0, 0.0, 1.0]])
print("Distances to red and blue:", squared_distances(example_pixel, example_centers))
Give each pixel a group
Do: read assign_pixels: calculate distances, then pick the smallest.
argmin means “which position has the smallest value?” Python counts from 0, so cluster 0 is the first group.
Check: do pixels choose the closest center, or does each center choose just one pixel? Explain to a partner.
def assign_pixels(points, centers):
distances = squared_distances(points, centers)
return distances.argmin(axis=1)
assert assign_pixels(example_pixel, example_centers).tolist() == [0]
print("Assignment works: our red test pixel joins cluster 0.")
5 · Mean means average
One idea: a center moves to the average of its group's pixels.
Two red values, 0.8 and 1.0, average to 0.9. That is the “means” in k-means.
Do: run the example. Point to members.mean(axis=0).
Say it: “Group by the nearest center. Move the center to the group's average.”
def update_centers(points, labels, old_centers):
new_centers = old_centers.copy()
for group_id in range(len(old_centers)):
members = points[labels == group_id]
if len(members) > 0:
new_centers[group_id] = members.mean(axis=0)
return new_centers
test_points = np.array([[0.8, 0.0, 0.0], [1.0, 0.2, 0.0]])
print("New center:", update_centers(test_points, np.array([0, 0]), example_centers[:1]))
6 · The k-means training loop
Do: run the loop. It chooses starting centers, groups the pixels, updates the averages, and goes again.
Notice: the printout says whether the groups settled or the iteration limit stopped us.
Check: what would happen if we grouped the pixels once but never moved the centers?
Inside the loop
Starting centers are spread-out pixels. Empty groups keep their old center. Every iteration saves its actual centers and assignments; the viewer replays those states.The k-means iteration loopOpen the Python code
#@title The k-means iteration loop
def initial_centers(points, k, seed):
rng = np.random.default_rng(seed)
chosen = [points[rng.integers(len(points))].copy()]
while len(chosen) < k:
nearest_distance = squared_distances(points, np.array(chosen)).min(axis=1)
chosen.append(points[nearest_distance.argmax()].copy())
return np.array(chosen)
def run_kmeans(points, k, seed=7, max_iterations=20):
centers = initial_centers(points, k, seed)
history = []
previous_labels = None
for iteration in range(max_iterations):
labels = assign_pixels(points, centers)
new_centers = update_centers(points, labels, centers)
changes = len(points) if previous_labels is None else int((labels != previous_labels).sum())
# Mean squared distance AFTER updating centers for these assignments.
loss = float(((points - new_centers[labels]) ** 2).sum(axis=1).mean())
history.append({"iteration": iteration + 1, "centers_before": centers.copy(),
"centers_after": new_centers.copy(), "labels": labels.copy(),
"loss": loss, "changed": changes})
if previous_labels is not None and changes == 0:
break
previous_labels = labels.copy()
centers = new_centers
return history
history = run_kmeans(X, K, SEED, MAX_ITERATIONS)
final = history[-1]
label_image = final["labels"].reshape(height, width)
cluster_settings = {"k": K, "position_weight": POSITION_WEIGHT, "seed": SEED,
"max_iterations": MAX_ITERATIONS, "cluster_side": CLUSTER_SIDE}
print(f"Saved {len(history)} iterations; {len(np.unique(final['labels']))} nonempty groups out of {K} requested.")
print("Stopped because:", "assignments stopped changing" if final["changed"] == 0 else "iteration limit reached")
7 · Catch the groups changing
Predict: will the first and last pictures have the same boundaries?
Do: run the viewer, then drag the iteration slider from start to finish. If the slider doesn't display, change STEP and rerun the viewer.
Notice: group colors change the way the photo is divided. They do not name the objects.
Black crosses show old centers; rings show updated centers. The dots plot shows red/green only.
Open the step viewerOpen the Python code
#@title Open the step viewer
def show_iteration(step, display_plot=True):
frame = history[int(step)]
labels = frame["labels"]
fig, axes = plt.subplots(1, 4, figsize=(16, 4))
axes[0].imshow(small_image); axes[0].set_title("1 · Original")
axes[1].imshow(palette[labels].reshape(height,width,3)); axes[1].set_title("2 · Assign pixels")
rebuilt = frame["centers_after"][labels, :3].reshape(height,width,3)
axes[2].imshow(np.clip(rebuilt,0,1)); axes[2].set_title("3 · Rebuild with group means")
ids = preview_ids
axes[3].scatter(X[ids,0], X[ids,1], c=palette[labels[ids]], s=5, alpha=.35)
before, after = frame["centers_before"], frame["centers_after"]
axes[3].scatter(before[:,0],before[:,1],marker="x",c="black",s=90,label="Before")
axes[3].scatter(after[:,0],after[:,1],facecolors="none",edgecolors="black",s=140,label="After")
for a,b in zip(before,after):
axes[3].annotate("",xy=b[:2],xytext=a[:2],arrowprops={"arrowstyle":"->","color":"black"})
axes[3].set(xlabel="Red",ylabel="Green",title="4 · Update centers")
axes[3].legend(fontsize=8)
for ax in axes[:3]: ax.axis("off")
fig.suptitle(f"Iteration {frame['iteration']} · {frame['changed']:,} assignments changed · mean squared distance {frame['loss']:.4f}")
plt.tight_layout()
if display_plot:
plt.show()
return fig
STEP = 0 # 0 is the first saved iteration. Try len(history) - 1 for the last one.
_ = show_iteration(min(max(STEP, 0), len(history)-1))
Move through the saved stepsOpen the Python code
#@title Move through the saved steps
# Optional slider. The STEP cell above still works if widgets do not display.
import ipywidgets as widgets
step_slider = widgets.IntSlider(value=0, min=0, max=len(history)-1, description="Iteration", continuous_update=False)
iteration_image = widgets.Image(format="png", layout=widgets.Layout(width="100%"))
def refresh_iteration(change):
# Render the actual saved algorithm state into the image widget.
# This avoids relying on nested notebook output capture.
fig = show_iteration(step_slider.value, display_plot=False)
buffer = io.BytesIO()
fig.savefig(buffer,format="png",bbox_inches="tight")
plt.close(fig)
iteration_image.value = buffer.getvalue()
step_slider.observe(refresh_iteration,names="value")
refresh_iteration(None)
display(step_slider, iteration_image)
Follow one pixel
Do: run the cell, then move PIXEL_X or PIXEL_Y to another part of the photo.
Notice: the smallest printed distance wins. This is the exact decision used in that iteration.
Check: could a pixel switch groups even though its own color never changes? What else is moving?
PIXEL_X, PIXEL_Y = 0.5, 0.5
pixel_x = int(np.clip(PIXEL_X,0,1) * (width-1))
pixel_y = int(np.clip(PIXEL_Y,0,1) * (height-1))
index = pixel_y * width + pixel_x
frame = history[min(max(STEP,0),len(history)-1)]
distances = squared_distances(X[index:index+1], frame["centers_before"])[0]
for group_id, value in enumerate(distances):
print(f"Cluster {group_id}: squared distance = {value:.5f}")
print("Winning cluster:", distances.argmin(), "· actual assignment:", frame["labels"][index])
fig, ax = plt.subplots(figsize=(5,4)); ax.imshow(small_image)
ax.plot(pixel_x,pixel_y,"r+",markersize=16); ax.axis("off"); plt.show()
A falling line is not “more correct”
One idea: the curve measures how close pixels are to their centers.
Do: follow the curve left to right. A lower value means tighter groups in our chosen measurements.
Check: does a lower curve prove the algorithm found the dog? Why not?
Compare this score only when the measurement weights stay the same. More groups are not automatically more useful.
Check how tightly the groups fitOpen the Python code
#@title Check how tightly the groups fit
fig, axes = plt.subplots(1,2,figsize=(10,3))
axes[0].plot([f["iteration"] for f in history],[f["loss"] for f in history],"o-")
axes[0].set(xlabel="Iteration",ylabel="Mean squared distance",title="Clustering objective")
counts = np.bincount(final["labels"],minlength=K)
axes[1].bar(range(K), counts, color=palette[:K])
axes[1].set(xlabel="Cluster ID",ylabel="Number of pixels",title="How large are the groups?")
plt.tight_layout(); plt.show()
8 · Remix one rule
Choose one. Predict first.
| Change | Try | Look for |
|---|---|---|
| Number of groups | K: 5 → 8 | More useful detail, or more pieces? |
| Use color only | POSITION_WEIGHT: 0.3 → 0 | Same color, far apart: one group? |
| Make location matter | POSITION_WEIGHT: 0.3 → 1 | Does one object split? |
| New photo | USE_UPLOAD = True in section 2 | What stops working? |
Do: rerun from the changed cell downward. Put the before/after side by side.
Tell a partner: “I changed ___. I expected ___. I saw ___.”
prediction = "Write your prediction before changing one setting."
observation = "After running, describe a visible change and a failure."
print("Prediction:", prediction)
print("Observation:", observation)
Concept switch · KNN asks the neighbors
Same K. Different job. In k-means, K counts groups. In KNN, K counts nearby examples.
Predict: this pixel looks pale. If its three closest labeled colors vote, which label wins?
Do: run the cell. Try NEIGHBORS = 1, then 5. Move the chosen point to the grass.
Notice: we supplied the labels “green,” “light,” and “dark.” KNN reuses those labels; k-means had none. These are color labels, not object names.
NEIGHBORS = 3
KNN_X, KNN_Y = 0.5, 0.5
# Nine color examples with labels we supplied.
known_colors = np.array([[.1,.5,.1],[.2,.7,.2],[.3,.5,.2],
[.8,.8,.8],[.95,.95,.9],[.7,.75,.8],
[.05,.05,.05],[.15,.2,.15],[.25,.2,.2]])
known_labels = np.array(["green"]*3 + ["light"]*3 + ["dark"]*3)
def knn_vote(colors, labels, query, neighbors=3):
if not isinstance(neighbors,int) or not 1 <= neighbors <= len(colors):
raise ValueError("Choose 1–9 neighbors.")
distances = ((colors-query)**2).sum(axis=1)
nearest_ids = np.argsort(distances,kind="stable")[:neighbors]
names,votes = np.unique(labels[nearest_ids],return_counts=True)
# A tied vote picks the alphabetically first label in this demo.
return str(names[votes.argmax()]),nearest_ids,dict(zip(names.tolist(),votes.tolist()))
qx = int(np.clip(KNN_X,0,1)*(width-1))
qy = int(np.clip(KNN_Y,0,1)*(height-1))
query_color = rgb[qy,qx]
knn_label, neighbor_ids, knn_votes = knn_vote(known_colors,known_labels,query_color,NEIGHBORS)
fig, axes = plt.subplots(1,NEIGHBORS+1,figsize=(2*(NEIGHBORS+1),2),squeeze=False)
for ax,color,title in zip(axes[0],[query_color,*known_colors[neighbor_ids]],
["Your pixel",*known_labels[neighbor_ids]]):
ax.imshow(np.tile(color,(12,12,1)));ax.set_title(title);ax.axis("off")
plt.tight_layout();plt.show()
print("Neighbor votes:",knn_votes,"→ predicted color label:",knn_label)
Concept switch · Regression predicts a number
One idea: a label says “light.” A number says “brightness ≈ 0.73.”
Predict: can a pixel's red value tell us its exact brightness?
Do: fit a straight-line trend to real pixels from this photo. Change RED_TO_TRY between 0 and 1.
Notice: dots scatter around the line. Red is useful, but green and blue also matter. Circles helped fit the line; crosses are pixels it didn't fit on.
This demo's brightness is the RGB average, not a full measure of human-perceived brightness.
RED_TO_TRY = 0.6
# Fit on some pixels; check on different pixels from the same photo.
if height*width < 4:
raise ValueError("Use a larger image for the regression experiment.")
regression_ids = np.random.default_rng(7).permutation(height*width)[:80]
fit_count = min(60,max(2,len(regression_ids)*3//4))
regression_pixels = rgb.reshape(-1,3)[regression_ids]
red_values = regression_pixels[:,0]
brightness_values = regression_pixels.mean(axis=1)
def fit_brightness_line(red, brightness):
if np.ptp(red) < 1e-8: # A single red value cannot define a useful slope.
return np.array([0.0,float(np.mean(brightness))])
return np.polyfit(red,brightness,1)
line = fit_brightness_line(red_values[:fit_count],brightness_values[:fit_count])
brightness_guess = float(np.polyval(line,np.clip(RED_TO_TRY,0,1)))
fig,ax = plt.subplots(figsize=(6,4))
ax.scatter(red_values[:fit_count],brightness_values[:fit_count],c=regression_pixels[:fit_count],edgecolors="#777",label="Used to fit")
ax.scatter(red_values[fit_count:],brightness_values[fit_count:],c=regression_pixels[fit_count:],marker="x",label="New pixels")
axis_values = np.linspace(0,1,100)
ax.plot(axis_values,np.polyval(line,axis_values),color="#315d4b",label="Learned trend")
ax.plot(np.clip(RED_TO_TRY,0,1),brightness_guess,"o",color="#b26d25",markersize=10,label="Your prediction")
ax.set(xlabel="Red value",ylabel="Brightness (RGB average)",title="Regression: predict a number")
ax.legend(fontsize=8);plt.tight_layout();plt.show()
print(f"Red {np.clip(RED_TO_TRY,0,1):.2f} → predicted brightness {brightness_guess:.2f}")
Quick check · Group, label, or number?
Say your answer before opening the reveal.
No labels. We want five pixel groups.
K-means. K is the number of groups; their centers move to the averages.We have labeled colors. Name a new color from nearby examples.
KNN classification. Nearby examples vote. K is the number of neighbors.We want a brightness value for a new pixel.
Regression. The target is a number. Our example uses a straight-line trend.Back to the mission: naming a color is easier than recognizing a dog. Next we borrow a model trained on labeled photos.
9 · A cluster is not an object
Predict: does one color group contain the whole dog?
Do: choose a point using SELECT_X and SELECT_Y (0 = left/top; 1 = right/bottom). The cell shows its cluster, then boxes around connected pieces.
Notice: one cluster can have several separate patches. One object can use several clusters.
Say it: “Clustering finds similar pixels. It doesn't know their names.”
Tiny patches are filtered out; we keep up to four larger regions for the next step.
SELECT_X, SELECT_Y = 0.5, 0.5
MIN_REGION_PIXELS = 40
MAX_REGIONS = 4
PADDING_FRACTION = 0.25
select_x = int(np.clip(SELECT_X,0,1)*(width-1))
select_y = int(np.clip(SELECT_Y,0,1)*(height-1))
selected_cluster = int(label_image[select_y,select_x])
selected_mask = label_image == selected_cluster
# This helper returns components in deterministic, largest-first order.
def find_regions(mask, min_pixels=40, max_regions=4):
components, _ = ndimage.label(mask) # Four-side connectivity.
sizes = np.bincount(components.ravel())
regions = []
for component_id, slices in enumerate(ndimage.find_objects(components),start=1):
if slices is None or sizes[component_id] < min_pixels:
continue
ys,xs = slices
regions.append({"component_id": component_id,"pixels": int(sizes[component_id]),
"box": (xs.start,ys.start,xs.stop,ys.stop)})
regions.sort(key=lambda r:(-r["pixels"],r["component_id"]))
return components, regions[:max_regions], len(regions)
components, regions, eligible_regions = find_regions(selected_mask,MIN_REGION_PIXELS,MAX_REGIONS)
print(f"Selected cluster {selected_cluster}: {int(selected_mask.sum()):,} pixels.")
print(f"Showing {len(regions)} of {eligible_regions} regions above the size filter.")
if not regions:
print("No region passed the size filter. Lower MIN_REGION_PIXELS or choose a different point.")
fig, axes = plt.subplots(1,3,figsize=(12,4))
axes[0].imshow(small_image); axes[0].plot(select_x,select_y,"r+",markersize=16)
axes[0].set_title("Chosen point")
axes[1].imshow(selected_mask,cmap="gray"); axes[1].set_title(f"Cluster {selected_cluster} mask")
axes[2].imshow(small_image); axes[2].set_title("Connected region proposals")
for number,region in enumerate(regions,start=1):
x1,y1,x2,y2 = region["box"]
axes[2].add_patch(Rectangle((x1,y1),x2-x1,y2-y1,fill=False,edgecolor="yellow",linewidth=2))
axes[2].text(x1,y1,str(number),color="black",bbox={"facecolor":"yellow","pad":2})
for ax in axes: ax.axis("off")
plt.tight_layout(); plt.show()
10 · Give the detector some context
One idea: a tiny patch of white fur is harder to recognize than a dog-shaped crop.
Do: look at each crop. Try PADDING_FRACTION = 0, then 0.5 in section 9 and rerun from there.
Notice: padding adds surroundings. It does not change which pixels belong to the region.
The detector sees crops of the original photo, with texture intact.
See the region cropsOpen the Python code
#@title See the region crops
def make_crops(image, components, regions, padding_fraction=0.25):
full_width, full_height = image.size
grid_height,grid_width = components.shape
region_crops = []
for region in regions:
x1,y1,x2,y2 = region["box"]
pad_x = max(2,round((x2-x1)*padding_fraction))
pad_y = max(2,round((y2-y1)*padding_fraction))
left = max(0,int(np.floor((x1-pad_x)/grid_width*full_width)))
top = max(0,int(np.floor((y1-pad_y)/grid_height*full_height)))
right = min(full_width,int(np.ceil((x2+pad_x)/grid_width*full_width)))
bottom = min(full_height,int(np.ceil((y2+pad_y)/grid_height*full_height)))
crop = image.crop((left,top,right,bottom))
mask_image = Image.fromarray((components==region["component_id"]).astype(np.uint8)*255)
full_mask = mask_image.resize(image.size,Image.Resampling.NEAREST)
crop_mask = np.array(full_mask.crop((left,top,right,bottom)))>0
region_crops.append({**region,"crop":crop,"mask":crop_mask,"origin":(left,top),
"crop_box":(left,top,right,bottom)})
return region_crops
region_crops = make_crops(source_image,components,regions,PADDING_FRACTION)
if region_crops:
fig, axes = plt.subplots(1,len(region_crops),figsize=(4*len(region_crops),4),squeeze=False)
for number,(ax,region) in enumerate(zip(axes[0],region_crops),start=1):
ax.imshow(region["crop"]); ax.set_title(f"Region {number} · {region['pixels']} cluster pixels"); ax.axis("off")
plt.tight_layout(); plt.show()
11 · Where do the names come from?
One idea: we are borrowing a model that already learned from labeled photos.
Do: load the detector. It downloads its weights once per fresh runtime.
Check: did our k-means loop learn the word “dog”? Where will that word come from now?
Model and setup details
We use pretrained SSDLite320 MobileNetV3 COCO weights, about 13.4 MB. It knows a fixed set of common categories and can miss objects. Inference stays in your Colab runtime. For a torchvision::nms import error, start a fresh standard Colab runtime. Model documentation.Load the pretrained detectorOpen the Python code
#@title Load the pretrained detector
if importlib.util.find_spec("torch") is None or importlib.util.find_spec("torchvision") is None:
subprocess.check_call([sys.executable,"-m","pip","install","-q",
"torch==2.6.0","torchvision==0.21.0",
"--index-url","https://download.pytorch.org/whl/cpu"])
import torch
import torchvision
from torchvision.models.detection import (ssdlite320_mobilenet_v3_large,
SSDLite320_MobileNet_V3_Large_Weights)
torch.set_num_threads(2) # Keep model setup and inference light on a CPU runtime.
weights = SSDLite320_MobileNet_V3_Large_Weights.COCO_V1
detector = ssdlite320_mobilenet_v3_large(weights=weights).eval().to("cpu")
preprocess = weights.transforms()
categories = weights.meta["categories"]
print("Loaded pretrained SSDLite · CPU inference")
print("torch",torch.__version__,"· torchvision",torchvision.__version__)
print("Some available names:",", ".join(n for n in categories if n not in ["N/A","__background__"])[:240])
12 · Ask what is inside
Do: run recognition. Read one proposed name, its score, and its region overlap.
Two filters:
- Score: is the detector's guess strong enough?
- Overlap: does its box cover enough of our chosen region?
Predict, then try: raise SCORE_THRESHOLD from 0.35 to 0.65. Which boxes disappear?
A high score can still be wrong. Overlap connects a box to a region; it doesn't label every pixel in that region.
SCORE_THRESHOLD = 0.35
MIN_REGION_OVERLAP = 0.10
if not 0 <= SCORE_THRESHOLD <= 1 or not 0 <= MIN_REGION_OVERLAP <= 1:
raise ValueError("Both thresholds must be between 0 and 1.")
def box_overlap(mask, box):
# Detector coordinates are in this crop's original pixel dimensions.
h,w = mask.shape
x1,y1,x2,y2 = box
left,top = max(0,int(np.floor(x1))),max(0,int(np.floor(y1)))
right,bottom = min(w,int(np.ceil(x2))),min(h,int(np.ceil(y2)))
if right<=left or bottom<=top:
return 0.0
return float(mask[top:bottom,left:right].mean())
def recognize_regions(region_crops, score_threshold, min_overlap):
reports = []
# One crop at a time bounds memory and keeps the code inspectable.
for number,region in enumerate(region_crops,start=1):
with torch.inference_mode():
prediction = detector([preprocess(region["crop"])])[0]
proposals = []
for box,label,score in zip(prediction["boxes"],prediction["labels"],prediction["scores"]):
score = float(score)
if score < score_threshold:
continue
box = box.cpu().numpy()
overlap = box_overlap(region["mask"],box)
proposals.append({"name":categories[int(label)],"score":score,
"box":box.tolist(),"region_overlap":overlap,
"accepted":overlap>=min_overlap})
reports.append({"region":number,"crop_box":region["crop_box"],
"component_id":region["component_id"],"proposals":proposals})
return reports
recognition_reports = recognize_regions(region_crops,SCORE_THRESHOLD,MIN_REGION_OVERLAP)
for report in recognition_reports:
print(f"\nRegion {report['region']}")
if not report["proposals"]:
print(" No object proposals above the score threshold.")
for proposal in report["proposals"]:
status = "KEEP" if proposal["accepted"] else "REJECT: low overlap"
print(f" {proposal['name']}: score {proposal['score']:.2f}, region overlap {proposal['region_overlap']:.0%} → {status}")
if not region_crops:
print("No crops to recognize. Choose another point or lower the region size filter in section 9.")
Check the box against the photo
Do: point at a box. Does its name fit what you can see?
Notice: the left panel shows our cluster. The right panel shows the detector's guesses. They answer different questions.
Overlapping crops can detect the same object twice; this is not a unique object count.
Check: name one reason the detector could miss something.
Check the boxesOpen the Python code
#@title Check the boxes
def show_recognition(save_path=None):
fig,axes = plt.subplots(1,2,figsize=(12,5))
dimmed = rgb.copy() * 0.25
dimmed[selected_mask] = rgb[selected_mask]
axes[0].imshow(dimmed); axes[0].set_title(f"Selected cluster {selected_cluster} · bright pixels")
axes[1].imshow(source_image); axes[1].set_title("Detector boxes associated with these regions")
accepted = 0
for report in recognition_reports:
left,top,_,_ = report["crop_box"]
for proposal in report["proposals"]:
if not proposal["accepted"]:
continue
x1,y1,x2,y2 = proposal["box"]
axes[1].add_patch(Rectangle((left+x1,top+y1),x2-x1,y2-y1,
fill=False,edgecolor="#eea92b",linewidth=2))
axes[1].text(left+x1,top+y1,f"R{report['region']} {proposal['name']} · {proposal['score']:.2f}",
fontsize=8,bbox={"facecolor":"#fff3c2","alpha":.9,"pad":2})
accepted += 1
if accepted == 0:
axes[1].text(.5,.05,"No boxes passed both filters",transform=axes[1].transAxes,
ha="center",bbox={"facecolor":"white","alpha":.9})
for ax in axes: ax.axis("off")
plt.tight_layout()
if save_path is not None:
fig.savefig(save_path,bbox_inches="tight")
plt.show()
return accepted
accepted_box_count = show_recognition()
print(accepted_box_count,"accepted crop-level detections (not a unique object count).")
13 · Find a useful mistake
Your challenge: find one good result and one wrong or missing guess.
Try another cluster, a second object, or a photo whose subject matches its background. Keep the settings with each result.
Discussion:
- “K-means grouped these pixels because ___.”
- “The object name came from ___.”
- “This result fooled the system because ___.”
Build challenge: recreate assign_pixels or update_centers in a new cell. Check it against the tiny examples before swapping it in.
my_explanation = "Replace this with your explanation of clustering vs recognition."
my_failure_case = "Describe a missing or incorrect box and how you checked it."
print(my_explanation)
print(my_failure_case)
14 · Save and submit your results
Do: run the cell to make a ZIP. Download it from Colab's Files panel, or use the optional download cell.
Keep your notebook copy too; that saves your code edits and explanations.
Exit check — no scrolling back:
- K-means: what moves to the average?
- KNN: where do the labels come from?
- Regression: are we predicting a label or a number?
The ZIP contains figures, settings, assignments, and detector proposals. Its images may show your photo.
Export the resultsOpen the Python code
#@title Export the results
import zipfile
output_dir = Path("image-clustering-results")
output_dir.mkdir(exist_ok=True)
Image.fromarray((palette[label_image]*255).astype(np.uint8)).save(output_dir/"clusters.png")
reconstructed = np.clip(final["centers_after"][final["labels"],:3].reshape(height,width,3),0,1)
Image.fromarray((reconstructed*255).astype(np.uint8)).save(output_dir/"mean-colors.png")
show_recognition(output_dir/"recognition.png")
# Save the clustering comparison separately from the detector figure.
fig,axes = plt.subplots(1,2,figsize=(10,4))
axes[0].imshow(small_image);axes[0].set_title("Original clustering image")
axes[1].imshow(palette[label_image]);axes[1].set_title("Final groups (colors are IDs)")
for ax in axes: ax.axis("off")
plt.tight_layout();fig.savefig(output_dir/"comparison.png",bbox_inches="tight");plt.close(fig)
record = {"format":"eloc-image-clustering-v1","image_name":image_name,"image_sha256":image_sha256,
"working_size":list(source_image.size),"clustering_size":[width,height],
"settings":cluster_settings,"iterations":[{"iteration":f["iteration"],"loss":f["loss"],
"changed":f["changed"]} for f in history],"labels":label_image.tolist(),
"selected_cluster":selected_cluster,"selection_point":[select_x,select_y],
"region_settings":{"min_pixels":MIN_REGION_PIXELS,"max_regions":MAX_REGIONS,"padding":PADDING_FRACTION},
"recognition_settings":{"model":"SSDLite320_MobileNet_V3_Large_COCO_V1",
"score_threshold":SCORE_THRESHOLD,"min_region_overlap":MIN_REGION_OVERLAP},
"recognition_reports":recognition_reports,"prediction":prediction if isinstance(prediction,str) else "",
"concepts":{"knn":{"neighbors":NEIGHBORS,"predicted_label":knn_label,"votes":knn_votes},
"regression":{"red":float(np.clip(RED_TO_TRY,0,1)),"predicted_brightness":brightness_guess}},
"observation":observation,"explanation":my_explanation,"failure_case":my_failure_case,
"versions":{"numpy":np.__version__,"torch":torch.__version__,"torchvision":torchvision.__version__}}
(output_dir/"experiment.json").write_text(json.dumps(record,indent=2))
with zipfile.ZipFile("image-clustering-results.zip","w",zipfile.ZIP_DEFLATED) as archive:
for path in sorted(output_dir.iterdir()): archive.write(path,arcname=path.name)
print("Created image-clustering-results.zip. Download it from Colab's Files panel.")
Download the ZIP (optional)Open the Python code
#@title Download the ZIP (optional)
# Optional: set True to open Colab's download prompt.
DOWNLOAD_RESULTS = False
if DOWNLOAD_RESULTS:
from google.colab import files
files.download("image-clustering-results.zip")
Need help or want to go deeper?
- Colab help
- Pixel-distance arrays
- Connected regions
- Object detector
- KNN: classification and regression
- Sample photo
Keep these: group/average/repeat · neighbors vote · trends predict numbers.
KNN can predict numbers too by averaging nearby targets. In our experiment, it votes on color labels. The object detector is a separate pretrained neural network.