Evaluation perplexity; lower is better.

A Basic Perplexity Measure

Train on FineWeb for however much you'd like, and evaluate on any subset you'd like (probably should be more specific than this...)

from datasets import load_dataset
dataset = load_dataset(
    "HuggingFaceFW/fineweb",
    split="train",
    streaming=True,
)

Participate

measurement = experiment.measure(checkpoint)

@measurement.evaluate(params=0, batch=1)
def evaluate(params, batch):
    ...

@measurement.aggregate
def aggregate(outputs):
    ...

score = measurement.run(checkpoint, batches)
measurement.submit("perplexity", score)

# Next steps? Getting started docs

Results

0 runs

No results.