alallo docs

Appearance

Evaluation report ​

What the alallo evaluation report is ​

The evaluation report is an optional add-on to a fine-tuning job. When the job has finished training, your fine-tuned model and the model it started from (the base model) both answer the same examples, which the fine-tuned model was not trained on. The report compares their answers with the answer in your data. If your file came from an API's logs, for example a file you used for OpenAI fine-tuning, the answer in your data is your old model's answer, so the report shows how close the new model gets to it.

The report is off unless you turn it on. Without it, nothing is held back from training and nothing is charged for it.

Turning on the alallo evaluation report ​

In New job, under Data, tick Evaluation report (optional, extra cost). Then choose the test examples:

  • Hold back 5% of my data (at most 200 examples): alallo sets aside 5% of your examples, at least 10 and at most 200, before training. These examples are never trained on. The same examples are held back if a job is retried.
  • Test file: choose another file you uploaded. The report uses up to 200 of its examples, and all of your training data is trained on.

Holding back needs enough data: if your file has too few examples to set 10 aside, alallo asks you to upload a separate test file.

Data the alallo evaluation report works with ​

The report needs a question and an answer in each example, so it works with:

  • Chat files (OpenAI or Mistral chat format): the last assistant message is the answer, and everything before it is the question.
  • Prompt/completion files: the completion is the answer.
  • Instruction/output files (Alpaca): the output is the answer.
  • Preference files (for DPO): the preferred answer is the answer.

Plain text data with a text template, and Hugging Face datasets, can't be evaluated yet: the box can't be ticked for them.

The price of the alallo evaluation report ​

The estimate shows the report on its own line, Evaluation report, in alallo tokens, with the number of examples. It is included in the job's price and reserved with it. The report costs much less than the training itself for most jobs, because each model only answers up to 200 examples.

You pay for the report only when you get it. If the report can't be made, for example because something went wrong after training finished, its tokens go back to your balance and the job shows that the report wasn't charged. If the job is cancelled or fails, the report is not made and not charged.

The report adds a few minutes after training to the job's estimated time.

What the alallo evaluation report shows ​

When the job has succeeded, the report appears in the job's detail in Jobs, under the model download:

  • Scores for the base model and your fine-tuned model:
    • Matches the answer word for word: the share of answers that equal the answer in your data, ignoring case, spaces and a final full stop. When your answers are short (classification or extraction, five words or fewer for most examples), this is the main score.
    • Similarity to the answer (ROUGE-L): how much of the wording of the answer in your data the model's answer has, in order, from 0% to 100%. This is the main score for free text.
    • Answer length: the average length of each model's answers, in tokens.
  • Speed: the time per answer when requests come one at a time, and the machine time to answer 1,000 requests like these when they are answered in batches.
  • Cost per 1,000 requests: enter your old API's price per million input and output tokens, and the hourly price of a machine you would run your model on, and the report works out the cost of 1,000 requests like these for both. These are estimates from the report's examples; your real traffic may differ.
  • Examples side by side: 20 examples, each with the answer in your data, the base model's answer and your fine-tuned model's answer.

Both models answer with the same settings (no sampling, so the answer is the most likely one), in the model's own chat format for chat data.

How long alallo keeps the evaluation report ​

The report contains examples from your data, so it is kept like your fine-tuned model: for 30 days after the job finishes, then deleted. [TBD: download the report as a file.]