AI CLASS LAB — Train, Tune & Test a Classifier

From sentiment analysis → TikTok captions → visual AI
Browser-only: Google Colab + Hugging Face. No local compiler. Use CPU runtime for the core lab.

Route / time

Part A (core, ~30–40 min): train a sentiment classifier, experiment with n-grams and models, measure performance, launch an app.
Part B (investigate, ~15–25 min): experience how features, targets, samples and evaluation change what the model learns.
Part C (optional, ~15–25 min): TikTok video popularity from caption/on-screen text, using one small monthly file, not the whole dataset.
Part D (optional, more time + model download): visual question answering and object recognition with pretrained models.

Student workflow: Run cells from the top. Try changing values. Record the effect of each change. The experiments are invitations, not prescribed results. Links: Rotten Tomatoes · TikTok · COCO VQA.

01 / Setup

Install libraries into the hosted Colab runtime. pip in Colab is not a local installation on your computer.

In [ ]: Python · cell 1

%pip -q install -U datasets scikit-learn pandas matplotlib gradio huggingface_hub pyarrow requests pillow

02 / Choose X and y

For sentiment analysis, X is the review text and y is its label: 0 = negative, 1 = positive. The splits are already supplied by the dataset; we will not tune on the final test split.

Why split? Training examples are for fitting weights. Validation examples guide choices. Test examples estimate how well the final choice generalizes.

In [ ]: Python · cell 2

from datasets import load_dataset
import pandas as pd
import numpy as np

dataset = load_dataset("cornell-movie-review-data/rotten_tomatoes")
train = dataset["train"].to_pandas()
valid = dataset["validation"].to_pandas()
test = dataset["test"].to_pandas()

print("Train / valid / test:", len(train), len(valid), len(test))
print("Labels:", dataset["train"].features["label"].names)
print(train.sample(4, random_state=7).to_string(index=False))

In [ ]: Python · cell 3

print("Training label proportions:")
print(train["label"].value_counts(normalize=True).sort_index())
print("Median review length:", int(train["text"].str.len().median()), "characters")

03 / What is an n-gram?

An n-gram is a sequence of n consecutive tokens. ngram_range=(1,1) uses single words, (1,2) adds adjacent word pairs, and (1,3) adds three-word phrases. More phrases can capture context but also increase vocabulary size and overfitting risk.

In [ ]: Python · cell 4

from sklearn.feature_extraction.text import CountVectorizer

examples = ["wait for it", "wait for the ending"]
for ngram_range in [(1, 1), (1, 2), (1, 3)]:
    v = CountVectorizer(ngram_range=ngram_range)
    v.fit(examples)
    print(ngram_range, "→", list(v.get_feature_names_out()))

04 / What is TF-IDF?

TF-IDF converts documents into numerical word/phrase features. A term gets weight when it appears in a document, but common terms across many documents get downweighted. TfidfVectorizer learns its vocabulary from training text only when used inside a Pipeline.

In [ ]: Python · cell 5

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(ngram_range=(1, 2), max_features=20)
small_matrix = vectorizer.fit_transform(train["text"].head(50))
print("Features:", list(vectorizer.get_feature_names_out()))
print("Matrix shape:", small_matrix.shape)

05 / Fit a first model

Logistic regression is a linear classifier: it learns a weight for each numeric feature and uses a logistic function to generate a probability for a binary label. Despite the name, it is for classification, not ordinary linear regression (which predicts a continuous number).

fit(X, y) learns weights using labeled training data; predict(X) assigns a class to new examples.

In [ ]: Python · cell 6

from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

model = Pipeline([
    ("tfidf", TfidfVectorizer(
        ngram_range=(1, 2),
        min_df=2,
        max_features=20_000
    )),
    ("clf", LogisticRegression(
        C=1.0,
        max_iter=1000,
        random_state=42
    ))
])

model.fit(train["text"], train["label"])
print(model.predict(["I loved it", "I wish I could get my time back"]))
print(model.predict_proba(["I loved it"]).round(3))

06 / Compare against a baseline

A baseline predicts without learning the relationship between text and sentiment. Here it always chooses the most frequent training label. A useful model should do more than simply guess the majority class.

Metrics: accuracy counts all correct predictions; F1 balances precision and recall; balanced accuracy averages recall across labels. The labels' imbalance can change which metric is most informative.

In [ ]: Python · cell 7

from sklearn.dummy import DummyClassifier
from sklearn.metrics import (
    accuracy_score, balanced_accuracy_score, f1_score,
    classification_report, ConfusionMatrixDisplay
)
import matplotlib.pyplot as plt

def metrics(y_true, y_pred):
    return {
        "accuracy": round(accuracy_score(y_true, y_pred), 3),
        "balanced_accuracy": round(balanced_accuracy_score(y_true, y_pred), 3),
        "f1": round(f1_score(y_true, y_pred, zero_division=0), 3)
    }

baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(train[["text"]], train["label"])
base_pred = baseline.predict(valid[["text"]])
model_pred = model.predict(valid["text"])
print("Majority baseline:", metrics(valid["label"], base_pred))
print("TF-IDF + logistic:", metrics(valid["label"], model_pred))
print(classification_report(valid["label"], model_pred, target_names=["negative", "positive"]))
ConfusionMatrixDisplay.from_predictions(valid["label"], model_pred)
plt.show()

07 / See what the model learned

With logistic regression and TF-IDF, positive coefficients support the positive class and negative coefficients support the negative class. They are associations in this training data, not causal explanations.

In [ ]: Python · cell 8

weights = model.named_steps["clf"].coef_[0]
terms = model.named_steps["tfidf"].get_feature_names_out()
order = np.argsort(weights)
print("Most negative features:", list(zip(terms[order[:12]], weights[order[:12]].round(2))))
print("Most positive features:", list(zip(terms[order[-12:]], weights[order[-12:]].round(2))))

08 / Dial reference: features

ngram_range: word sequence length, e.g. (1,1), (1,2), (1,3).
max_features: cap the number of vocabulary features, e.g. 5,000 vs 30,000.
min_df: keep terms seen in at least N training documents, e.g. 1 vs 3.
max_df: discard terms seen in an overly large fraction of documents, e.g. 0.95.
sublinear_tf: use 1 + log(tf) instead of raw term frequency.
analyzer: 'word' vs 'char_wb'; character groups can help with spelling variants.

These change X, the representation supplied to the learner, before fitting the classifier.

09 / Dial reference: models

LogisticRegression(C=...): controls regularization (smaller C = stronger constraints). Probability output with predict_proba.
class_weight='balanced': upweights minority training labels.
max_iter: optimizer's iteration limit; increasing may resolve convergence warnings, but does not guarantee better accuracy.
MultinomialNB(alpha=...): a different algorithm, smoothing helps prevent zero-frequency problems.
LinearSVC(C=...): linear support vector classifier, normally returns scores rather than probabilities.
random_state: makes supported randomized procedures reproducible; it is not an accuracy dial.

10 / Tune the text representation

Keep the underlying training and validation sets fixed, change the vectorizer settings, then compare validation performance. Edit the list to add your own combinations.

In [ ]: Python · cell 9

feature_settings = [
    {"ngram_range": (1, 1), "max_features": 5000, "min_df": 2},
    {"ngram_range": (1, 2), "max_features": 20000, "min_df": 2},
    {"ngram_range": (1, 3), "max_features": 30000, "min_df": 3},
    {"ngram_range": (1, 2), "max_features": 10000, "min_df": 5},
]
feature_rows = []
for settings in feature_settings:
    candidate = Pipeline([
        ("tfidf", TfidfVectorizer(**settings)),
        ("clf", LogisticRegression(C=1.0, max_iter=1000))
    ])
    candidate.fit(train["text"], train["label"])
    result = metrics(valid["label"], candidate.predict(valid["text"]))
    feature_rows.append({"settings": str(settings), **result})

pd.DataFrame(feature_rows).sort_values("f1", ascending=False)

11 / Compare algorithms and regularization

Only the classifier changes here, so the same text representation can be used with three different learners. For a stricter comparison, also tune their respective parameters.

In [ ]: Python · cell 10

from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import LinearSVC

learners = {
    "LogReg C=0.1": LogisticRegression(C=0.1, max_iter=1000),
    "LogReg C=1": LogisticRegression(C=1.0, max_iter=1000),
    "LogReg C=10": LogisticRegression(C=10.0, max_iter=1000),
    "Naive Bayes": MultinomialNB(alpha=1.0),
    "Linear SVC": LinearSVC(C=1.0, max_iter=2000),
}
model_rows = []
for name, learner in learners.items():
    candidate = Pipeline([
        ("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=2)),
        ("clf", learner)
    ])
    candidate.fit(train["text"], train["label"])
    model_rows.append({"learner": name,
                       **metrics(valid["label"], candidate.predict(valid["text"]))})

pd.DataFrame(model_rows).sort_values("f1", ascending=False)

12 / What do we mean by designing a model?

There are at least five levers: features (what information enters X), target (what does y mean), sampling (which examples are learned), evaluation (what counts as generalization), and fitting (which algorithm/parameters).

The next experiments isolate them. You may find that the bigger changes do not come from a new C value.

13 / Experience sample size

Keep the same validation set and model. Train on 10%, 25%, 50%, then 100% of the training rows. Smaller samples can train faster, but change what patterns the model gets to see.

In [ ]: Python · cell 11

sample_rows = []
for fraction in [0.10, 0.25, 0.50, 1.0]:
    subset = train.sample(frac=fraction, random_state=42)
    candidate = Pipeline([
        ("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=2)),
        ("clf", LogisticRegression(C=1.0, max_iter=1000))
    ])
    candidate.fit(subset["text"], subset["label"])
    sample_rows.append({"training_rows": len(subset),
                        **metrics(valid["label"], candidate.predict(valid["text"]))})

pd.DataFrame(sample_rows)

14 / Experience feature selection

Ask what evidence does the model need? Compare normal text with text where negation words were stripped. The algorithm and split stay fixed. The intervention is intentionally artificial: it is a diagnostic, not recommended preprocessing.

In [ ]: Python · cell 12

import re

def strip_negations(text):
    return re.sub(r"\b(no|not|never|cannot|nothing)\b|n't\b", "", text, flags=re.I)

normal_text = train["text"]
redacted_text = train["text"].map(strip_negations)
valid_redacted = valid["text"].map(strip_negations)

for name, tr_x, va_x in [
    ("Original text", normal_text, valid["text"]),
    ("Negations removed", redacted_text, valid_redacted),
]:
    m = Pipeline([
        ("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=2)),
        ("clf", LogisticRegression(max_iter=1000))
    ])
    m.fit(tr_x, train["label"])
    print(name, metrics(valid["label"], m.predict(va_x)))

15 / Experience target definition

The target decides what question is being answered. Below the labels become “long review vs short review,” using the training set's median character length as a cutoff. The text stays unchanged; the question changes. It is intentionally a simple pedagogical target, not a useful real-world product.

In [ ]: Python · cell 13

length_cutoff = train["text"].str.len().median()
y_train_length = (train["text"].str.len() > length_cutoff).astype(int)
y_valid_length = (valid["text"].str.len() > length_cutoff).astype(int)

length_classifier = Pipeline([
    ("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=2)),
    ("clf", LogisticRegression(max_iter=1000))
])
length_classifier.fit(train["text"], y_train_length)
print("Target: review length >", length_cutoff, "characters")
print(metrics(y_valid_length, length_classifier.predict(valid["text"])))
print("Positive-label fraction:", round(y_train_length.mean(), 3))

16 / Experience evaluation leakage

Do not run this approach in a real project. We intentionally append the answer to the text in both training and validation, and watch performance inflate. This reveals why a model can appear excellent without learning useful transferable structure.

Other examples: evaluating repeatedly on the final test split; random splitting when future samples differ from past ones; using post-publication engagement counts to predict engagement.

In [ ]: Python · cell 14

def attach_answer(df):
    return [f"{text} label_hint_{'positive' if label == 1 else 'negative'}"
            for text, label in zip(df["text"], df["label"])]

leaky = Pipeline([
    ("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=2)),
    ("clf", LogisticRegression(max_iter=1000))
])
leaky.fit(attach_answer(train), train["label"])
print("Normal validation:", metrics(valid["label"], model.predict(valid["text"])))
print("Leaked validation:", metrics(valid["label"],
                                   leaky.predict(attach_answer(valid))))

17 / Optional: systematic parameter search

A small GridSearchCV searches combinations using cross-validation inside the training split. Here validation remains a separate check; the test set stays untouched. A large grid is computationally expensive—start small.

In [ ]: Python · cell 15

from sklearn.model_selection import GridSearchCV

grid = GridSearchCV(
    estimator=Pipeline([
        ("tfidf", TfidfVectorizer()),
        ("clf", LogisticRegression(max_iter=1000))
    ]),
    param_grid={
        "tfidf__ngram_range": [(1, 1), (1, 2)],
        "tfidf__min_df": [2, 4],
        "clf__C": [0.1, 1.0, 10.0]
    },
    scoring="f1",
    cv=3,
    n_jobs=-1
)
grid.fit(train["text"], train["label"])
print("Best CV settings:", grid.best_params_)
print("Validation:", metrics(valid["label"],
                            grid.best_estimator_.predict(valid["text"])))

18 / Decision threshold

Logistic regression returns estimated probabilities. Classifying at 0.5 is a choice; higher thresholds usually increase precision while reducing recall. The best threshold depends on what mistakes cost in your use case.

In [ ]: Python · cell 16

from sklearn.metrics import precision_score, recall_score

prob_positive = grid.best_estimator_.predict_proba(valid["text"])[:, 1]
for threshold in [0.3, 0.5, 0.7]:
    predicted = (prob_positive >= threshold).astype(int)
    print("threshold", threshold,
          "precision", round(precision_score(valid["label"], predicted, zero_division=0), 3),
          "recall", round(recall_score(valid["label"], predicted, zero_division=0), 3))

19 / One final, untouched test

After you choose features and model using training/CV/validation, measure the final test only once. Note that rerunning many versions on the same test eventually turns it into another validation set.

In [ ]: Python · cell 17

final_model = grid.best_estimator_
final_predictions = final_model.predict(test["text"])
print("Final held-out test:", metrics(test["label"], final_predictions))
print(classification_report(test["label"], final_predictions,
                            target_names=["negative", "positive"]))

20 / Make a browser app

Gradio produces a temporary link served by your Colab session. Anyone with the link may be able to access it while it is running; don't upload private text.

Optional: use share=False if you want a local-in-Colab preview only.

In [ ]: Python · cell 18

import gradio as gr

def classify_review(text):
    values = final_model.predict_proba([text])[0]
    return {"Negative": float(values[0]), "Positive": float(values[1])}

review_app = gr.Interface(
    fn=classify_review,
    inputs=gr.Textbox(lines=3, label="Write a review"),
    outputs=gr.Label(num_top_classes=2),
    title="My sentiment classifier",
    examples=["I loved this movie", "The story was a complete mess"]
)
review_app.launch(share=True)

PART C / Optional: TikTok success classifier

Keep the TikTok idea. datasocial/tiktok-5.6B-videos contains caption, on-screen text, hashtags, views, and observation timestamps. The entire collection is ~460 GB and not appropriate to download in class.

Instead the following example uses 2018-01.parquet (~13.5 MB). This is a historical, potentially unrepresentative monthly slice. We are predicting observed relative popularity in that sample, not guaranteeing future TikTok virality. The released data is CC BY-NC 4.0.

Target y: top 20% of training-period views (threshold determined from train only). Features X: caption vs on-screen text vs both. Evaluation: chronological holdout. Avoid leakage: views/likes/shares/engagement must never appear in X. Stats were recorded at varying ages; filtering one month reduces but does not eliminate this limitation.

In [ ]: Python · cell 19

from huggingface_hub import hf_hub_download

monthly_path = hf_hub_download(
    repo_id="datasocial/tiktok-5.6B-videos",
    repo_type="dataset",
    filename="data/2018-01.parquet"
)
keep_columns = ["posted_at", "stats_updated_at", "caption", "on_screen_text",
                "views", "type", "language"]
tt = pd.read_parquet(monthly_path, columns=keep_columns)
print("Loaded rows:", len(tt))
print(tt[keep_columns].head(3).to_string(index=False))

In [ ]: Python · cell 20

MAX_ROWS = 12_000  # tweak down for a faster CPU experiment

tt["posted_at"] = pd.to_datetime(tt["posted_at"], utc=True, errors="coerce")
tt["stats_updated_at"] = pd.to_datetime(tt["stats_updated_at"], utc=True, errors="coerce")
tt["caption"] = tt["caption"].fillna("").astype(str)
tt = tt[(tt["type"] == "video") & tt["posted_at"].notna()
        & (tt["views"] > 0) & (tt["caption"].str.len() >= 5)].copy()

def join_screen_text(value):
    if isinstance(value, (list, tuple, np.ndarray)):
        return " ".join(str(item) for item in value if item is not None)
    return ""

tt["screen_text"] = tt["on_screen_text"].apply(join_screen_text)
tt["combined"] = tt["caption"] + " " + tt["screen_text"]
if len(tt) > MAX_ROWS:
    tt = tt.sample(n=MAX_ROWS, random_state=42)
tt = tt.sort_values("posted_at").reset_index(drop=True)
assert len(tt) >= 100, "Not enough valid rows; adjust filters or month."
print("Usable rows:", len(tt))
print("Has on-screen text:", int(tt["screen_text"].str.strip().ne("").sum()))

In [ ]: Python · cell 21

n = len(tt)
tr = tt.iloc[:int(0.6*n)].copy()
va = tt.iloc[int(0.6*n):int(0.8*n)].copy()
te = tt.iloc[int(0.8*n):].copy()

SUCCESS_QUANTILE = 0.80  # try 0.70, 0.90
view_cutoff = tr["views"].quantile(SUCCESS_QUANTILE)
for split in (tr, va, te):
    split["success"] = (split["views"] >= view_cutoff).astype(int)

print("Threshold (train only):", view_cutoff, "views")
print("Success fractions (train/valid/test):",
      [round(x["success"].mean(), 3) for x in (tr, va, te)])

In [ ]: Python · cell 22

from sklearn.metrics import average_precision_score

experiments = []
tiktok_models = {}
for name, column in [
    ("caption", "caption"),
    ("on-screen", "screen_text"),
    ("both", "combined")
]:
    if tr[column].str.strip().eq("").all():
        print("Skipping", name, "(no text in this slice)")
        continue
    candidate = Pipeline([
        ("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=2,
                                   max_features=20_000)),
        ("clf", LogisticRegression(C=1.0, class_weight="balanced",
                                    max_iter=1000))
    ])
    candidate.fit(tr[column], tr["success"])
    p = candidate.predict_proba(va[column])[:, 1]
    predicted = (p >= 0.5).astype(int)
    experiments.append({"features": name,
                        "average_precision": round(average_precision_score(va["success"], p), 3),
                        **metrics(va["success"], predicted)})
    tiktok_models[name] = candidate
print("Random-ranking AP baseline ≈ positive fraction:", round(va["success"].mean(), 3))
pd.DataFrame(experiments).sort_values("average_precision", ascending=False)

In [ ]: Python · cell 23

# Choose a configuration from VALIDATION before viewing TEST.
SELECTED_FEATURES = "caption"  # change to "on-screen" or "both" if available
selected_column = {"caption": "caption", "on-screen": "screen_text",
                   "both": "combined"}[SELECTED_FEATURES]
chosen = tiktok_models[SELECTED_FEATURES]
final_p = chosen.predict_proba(te[selected_column])[:, 1]
print("TikTok TEST average precision:",
      round(average_precision_score(te["success"], final_p), 3))
print("Top-20% threshold from train:", view_cutoff)
print("Note: not causal, not normalized to fixed video age.")

TikTok questions to investigate

Target: change SUCCESS_QUANTILE from 0.80 to 0.90. Is the class balance now harder? Why are F1 and average precision sensitive to this choice?
Features: compare caption, on-screen text, both. Which field is actually populated in this monthly archive?
Sampling: compare one old month with a different month without downloading the entire archive. How representative is it of present-day TikTok?
Evaluation: compare random and chronological holdouts (note time and creator duplicates). Why might apparent performance change?
Potential leakage: would views as an input make a dishonest success detector? What about stats_updated_at, late-added captions, or not_recommended_for_fyp?
Limits: follower count, visuals, music and recommendation effects aren't in our text features; the dataset doesn't measure popularity at an equal age for all posts.

PART D / Visual extension: image classification ≠ VQA

The fawern/visual-question-answering-coco dataset has 217 rows with three columns: image_url, question, answer. That's a small set of question–answer examples, not a labeled object-classification dataset. We will evaluate a pretrained model, rather than train visual question answering from scratch on 217 examples.

Image classification: image → a category, e.g. coffee mug.
Visual question answering (VQA): image + question → text, e.g. What am I holding? → a mug.
Object detection: image → locations and labels for multiple objects.

The next model downloads can be large; run this part if the class has time and bandwidth.

In [ ]: Python · cell 24

from datasets import load_dataset

vqa_data = load_dataset("fawern/visual-question-answering-coco", split="train")
print("Rows:", len(vqa_data))
print("Columns:", vqa_data.column_names)
print(vqa_data[0])

In [ ]: Python · cell 25

import requests
from io import BytesIO
from PIL import Image
from IPython.display import display

row = vqa_data[0]
preview_image = None
for url in [row["image_url"].replace("http://", "https://"), row["image_url"]]:
    try:
        response = requests.get(url, timeout=20)
        response.raise_for_status()
        preview_image = Image.open(BytesIO(response.content)).convert("RGB")
        break
    except Exception as error:
        print("Could not fetch", url, str(error)[:110])

print("Question:", row["question"])
print("Reference answer:", row["answer"])
if preview_image is not None:
    display(preview_image)
else:
    print("COCO image host was unavailable. You can still upload your own image below.")

Run an already pretrained VQA model

This step performs inference, not fitting. Salesforce/blip-vqa-base provides existing vision-language weights; we are not teaching it new labels from the 217-row dataset. Natural-language answers can be correct despite not matching the provided reference word-for-word.

In [ ]: Python · cell 26

%pip -q install -U "transformers>=4.46,<5" accelerate

from transformers import pipeline

vqa_model = pipeline(
    task="visual-question-answering",
    model="Salesforce/blip-vqa-base",
    device=-1  # CPU; switch to 0 if a GPU runtime is available
)
if preview_image is not None:
    print(vqa_model(image=preview_image, question=row["question"]))

In [ ]: Python · cell 27

def ask_about_image(image, question):
    if image is None or not question.strip():
        return "Upload a photo and enter a question."
    output = vqa_model(image=image, question=question)
    return output[0]["answer"]

visual_app = gr.Interface(
    fn=ask_about_image,
    inputs=[gr.Image(type="pil", label="Photo or camera"),
            gr.Textbox(value="What object is this?", label="Question")],
    outputs=gr.Textbox(label="Answer"),
    title="Ask a question about an image"
)
visual_app.launch(share=True)

Optional: run a pretrained image classifier

A separate ImageNet image classifier predicts categories without a question. The COCO VQA dataset is not its training set. Try the same photo in both interfaces and compare the type of answer.

In [ ]: Python · cell 28

image_model = pipeline(
    task="image-classification",
    model="google/vit-base-patch16-224",
    device=-1
)

def identify_image(image):
    if image is None:
        return {}
    return {item["label"]: float(item["score"])
            for item in image_model(image, top_k=5)}

gr.Interface(
    fn=identify_image,
    inputs=gr.Image(type="pil", label="Upload or photograph an object"),
    outputs=gr.Label(num_top_classes=5),
    title="What object am I looking at?"
).launch(share=True)

Wrap-up / What to show your classmates

Choose one controlled change and report: (1) the question/target, (2) training and validation sample sizes, (3) which feature or algorithm parameter changed, (4) metric + baseline, (5) one failure case, and (6) what you'd try next.

If you use TikTok, explain why the label is only observed popularity and not proof that your text causes success. If you use VQA, distinguish using a pretrained model from training your own.

Links / reference

This notebook is a teaching template, not a benchmark or a trained prediction service. Results depend on the data snapshot and runtime.

Earlier extended notebook