Prediction: check held-out errors and training-only preparation

Keep test observations out of data preparation

Training observations are used to prepare and fit the model. Held-out observations are reserved for checking predictions. If missing-value means are calculated using the held-out observations, information from the test data has already entered the model-building process.

The reference therefore reconstructs which rows belong to each split and repeats preparation using training rows only. Both calculations must predict the same held-out cases and use the same comparison model and loss definition. Loss means a measure of prediction error. Fitting once on all observations would not check this procedure.

Which software produced these results?

The table identifies the saved QuickPLS prediction output and the independent packages used to reconstruct the reference calculations. This is a check of those saved results, not a new prediction run in the installed application.

On a small screen, swipe the table sideways to see every column.

Check PLSpredict and the predictive-ability comparison

PLSpredict evaluates predictions for observations left out during fitting. The cross-validated predictive ability test (CVPAT) compares prediction losses with a reference predictor. Both need the preparation, data assignments and loss definitions to match.

The two-block example uses 128 rows, 5 folds with 2 repeats, and a holdout setup. A fold is a part of the data temporarily held out while the other parts train the model. The reference reconstructs 11 data assignments.

• The reference reads the exported sum of squared loss differences as uncentred. It squares each PLS-minus-reference loss difference and adds the results without first subtracting the average difference. This corrects the reference interpretation of that output; it does not describe a new change to the application.
• This check does not include a new prediction run in the installed app.

Read prediction errors separately from software differences

The rows show squared-loss summaries for held-out cases. Indicator-average benchmark squared loss refers to the IA predictor; Linear-model benchmark squared loss refers to the LM predictor. Model squared loss is the PLS prediction error. Squared loss is the squared gap between an observed value and its prediction.

For the first displayed case, the PLS loss is larger than both reference losses. For the second, it is smaller. Both patterns agree across the calculations. A passing software comparison therefore does not mean PLS predicts better for every case, and these selected rows alone do not establish the CVPAT test result.

The table displays 15 significant digits. Absolute differences were calculated before this display rounding. The download keeps the saved precision and the matched loss definitions.

On a small screen, swipe the table sideways to see every column.

Error limits for matched prediction outputs

The error limits apply after matching the data assignments, training-only preparation and prediction-loss definitions. Exact checks and missing-result checks have their own rules; they are not additional independent prediction studies.

abs(native - aligned_reference) <= absolute_tolerance + relative_tolerance * abs(aligned_reference)

In this formula, native is the QuickPLS value and aligned_reference is the reference value expressed using the same definition. abs means absolute value, so the sign does not affect the size of the difference. Absolute tolerance is a fixed allowance. Relative tolerance adds an allowance that grows with the size of the reference value. A separate check verifies whether a result is available or missing where expected; it does not estimate another effect.

On a small screen, swipe the table sideways to see every column.

What these checks tell you

The comparison checks whether the stated prediction procedure is reproduced, including its preparation and errors on held-out cases. It does not promise that a model will predict well on a different population.

Use the complete prediction results to assess predictive performance. The count of compared values includes several outputs from the same splits and cases, rather than separate studies. Display rounding does not replace the saved numerical precision.

• Keep the fold assignments, training-only means, reference predictor and loss definition together when describing this result. Changing one changes the comparison.

Inspect the comparison files

The CSV lets you inspect prediction-loss values and their differences. The JSON report records the reconstructed assignments, preparation choices, reference calculations and software versions.

Repeating the comparison requires the original data, split settings, model, preparation rules and software versions. The downloads report completed tests and do not include the original datasets or application source code.

• Understand the comparison rules ↗
• Find checks for another method ↗