Carry out the manyROC analysis with cross-validation

Do the ROC analysis (roc_analysis) with cross-validation for each column of x (see also: roc_manyroc).

roc_manyroc_cv(
  x,
  gr,
  optimize_by = "bac",
  cvo = cvo_create_folds(x, gr, seeds = seeds, kind = kind),
  seeds = NA_real_,
  kind = NULL
)

Arguments

x	A numeric matrix, a data frame, a `hyperSpec` object or another type of object, convertible to a numeric matrix.
gr	Either a string (scalar, `character(1)`) or a `factor` variable (a vector) which defines groups in `x`.
optimize_by	(`string(1)`) A string with the name of classification performance measure to use. Currently available options: `"bac"` - for balanced accuracy (mean of sensitivity and specificity); `"kappa"` - for Cohens kappa; `"youden"` - for Youden's index;
cvo	a cross-validation object (cvo), created with function `cvo_create_folds`, caret `createFolds` or similar.
seeds	(`NA_real_` \| `NULL` \| vector of integers) Seeds for random number generator for each repetition. If `seeds = NA_real_` (default), no seeds are set, parameter `kind` is also ignored. If `seeds = NULL` random seeds are generated automatically and registered in attribute `"seeds"`. If numeric vector, then these seeds will be used for each repetition of cross-validation. If the number of repetitions is greater than the number of provided seeds, additional seeds are generated and added to the vector. The first seed will be used to ensure reproducibility of the randomly generated seeds. For more information about random number generation see `set.seed`.
kind	(`NULL` \| `character`) The kind of (pseudo)random number generator. Default is `NULL`, which selects the currently-used generator (including that used in the previous session if the workspace has been restored): if no generator has been used it selects `"default"`. Generator `"L'Ecuyer-CMRG"` is recommended if package parallel is used for for parallel computing. In this case each seed should have 6 elements neither the first three nor the last three should be all zero. More information at `set.seed`.

Value

A data frame with results. The object also inferits from class manyroc_cv (for displaying purposes).

Author

Vilmantas Gegzna

Examples


library(manyROC)


# --- For numeric vectors objects ---

data(PlantGrowth)

set.seed(123456)

roc_manyroc_cv(PlantGrowth$weight, PlantGrowth$group)
#>          set       fold compared_groups feature median_neg cutoff median_pos
#> 1   training Rep1_Fold1   ctrl vs. trt1       1       4.88   4.37       4.37
#> 2   training Rep1_Fold1   ctrl vs. trt2       1       4.88   5.21       5.33
#> 3   training Rep1_Fold1   trt1 vs. trt2       1       4.37   4.87       5.33
#> 4   training Rep1_Fold2   ctrl vs. trt1       1       4.88   4.37       4.55
#> ...      ...        ...             ...     ...        ...    ...        ...
#> 27      test Rep1_Fold4   trt1 vs. trt2       1         NA      5         NA
#> 28      test Rep1_Fold5   ctrl vs. trt1       1         NA   4.21         NA
#> 29      test Rep1_Fold5   ctrl vs. trt2       1         NA   5.22         NA
#> 30      test Rep1_Fold5   trt1 vs. trt2       1         NA    4.9         NA
#>                                                                             
#>      tp  fn  fp  tn sens spec  ppv  npv  bac youden kappa  auc
#> 1     5   3   1   7 0.62 0.88 0.83 0.70 0.75   0.50  0.50 0.73
#> 2     6   2   2   6 0.75 0.75 0.75 0.75 0.75   0.50  0.50 0.78
#> 3     8   0   1   7 1.00 0.88 0.89 1.00 0.94   0.88  0.88 0.91
#> 4     4   4   1   7 0.50 0.88 0.80 0.64 0.69   0.38  0.38 0.59
#> ... ... ... ... ...  ...  ...  ...  ...  ...    ...   ...  ...
#> 27    1   1   0   2 0.50 1.00 1.00 0.67 0.75   0.50  0.50   NA
#> 28    0   2   1   1 0.00 0.50 0.00 0.33 0.25  -0.50 -0.50   NA
#> 29    2   0   0   2 1.00 1.00 1.00 1.00 1.00   1.00  1.00   NA
#> 30    2   0   0   2 1.00 1.00 1.00 1.00 1.00   1.00  1.00   NA
#>                                                               

roc_manyroc_cv(PlantGrowth$weight, gl(2, 1, 30))
#>         set       fold compared_groups feature median_neg cutoff median_pos tp
#> 1  training Rep1_Fold1         1 vs. 2       1       5.25   5.27       4.91 10
#> 2  training Rep1_Fold2         1 vs. 2       1       5.17   4.09          5  3
#> 3  training Rep1_Fold3         1 vs. 2       1       5.25   5.12       5.13  7
#> 4  training Rep1_Fold4         1 vs. 2       1       5.25   5.48       5.13 10
#> 5  training Rep1_Fold5         1 vs. 2       1       5.17   3.66       5.01  2
#> 6      test Rep1_Fold1         1 vs. 2       1         NA   5.27         NA  1
#> 7      test Rep1_Fold2         1 vs. 2       1         NA   4.09         NA  0
#> 8      test Rep1_Fold3         1 vs. 2       1         NA   5.12         NA  2
#> 9      test Rep1_Fold4         1 vs. 2       1         NA   5.48         NA  2
#> 10     test Rep1_Fold5         1 vs. 2       1         NA   3.66         NA  0
#>    fn fp tn sens spec  ppv  npv  bac youden kappa  auc
#> 1   2  6  6 0.83 0.50 0.62 0.75 0.67   0.33  0.33 0.65
#> 2   9  0 12 0.25 1.00 1.00 0.57 0.62   0.25  0.25 0.57
#> 3   5  4  8 0.58 0.67 0.64 0.62 0.62   0.25  0.25 0.59
#> 4   2  7  5 0.83 0.42 0.59 0.71 0.62   0.25  0.25 0.57
#> 5  10  0 12 0.17 1.00 1.00 0.55 0.58   0.17  0.17 0.52
#> 6   2  2  1 0.33 0.33 0.33 0.33 0.33  -0.33 -0.33   NA
#> 7   3  0  3 0.00 1.00  NaN 0.50 0.50   0.00  0.00   NA
#> 8   1  2  1 0.67 0.33 0.50 0.50 0.50   0.00  0.00   NA
#> 9   1  3  0 0.67 0.00 0.40 0.00 0.33  -0.33 -0.33   NA
#> 10  3  0  3 0.00 1.00  NaN 0.50 0.50   0.00  0.00   NA