Introduction

This report checks if the status of packages on CRAN are due to intermittent failures.

Failures defined as warnings, notes or errors without change on:

R version used (if not stable the same svn snapshot)
The package version (Note that CRAN might modify a package without changing the version)
Their dependencies

Reasons of these failures might be because the packages depend on:

Random generation numbers
Flacky external resources
Other ?

Why is this important?

Because package maintainers of dependencies of that package, R core and CRAN team need to check if the failures are false positives.

This report started because it was suggested as something that the R-repositories working group could help the CRAN team.

Retrieve data

It makes use of tools::CRAN_check_results to retrieve the data.

library("dplyr")
library("tools", include.only = c("package_dependencies", "CRAN_check_results"))
library("flextable", include.only = c("flextable", "autofit"))
# Use a LOCAL environment to check if files can be overwritten on my computer
local_build <- as.logical(Sys.getenv("LOCAL", "FALSE"))
yc <- readRDS("today.RDS")
tc <- CRAN_check_results()
# Added 2023/03/09: sometimes some flavors are reported without status: Omit those
tc <- tc[!is.na(tc$Status),]
if (!interactive() && !local_build) {
  message("Saving today's file.")
  saveRDS(tc, file = "today.RDS")
}

The checks are from multiple flavors release, devel, old release and patched on multiple machines and configurations.

old_flavors <- readRDS("flavors.RDS")
flavors <- unique(tc$Flavor)
# One flavor now present in all is the r-devel-windows-x86_64: skip
flavors <- setdiff(flavors, "r-devel-windows-x86_64")
proto <- data.frame(r_version = character(),
                    os = character(),
                    architecture = character(),
                    other = character())
flavors_df <- strcapture(
  pattern = "r-([[:alnum:]]+)-([[:alnum:]]+)-([[:alnum:]_\\+]+)-?(.*)", 
  x = flavors,
  proto = proto)

# Extract R version used and svn id
h <- "https://www.r-project.org/nosvn/R.check/%s/ggplot2-00check.html"
links <- sprintf(h, flavors)
extract_revision <- function(x) {
  r <- readLines(x, 12)[12]
  version <- strcapture(pattern = "([[:digit:]]\\.[[:digit:]]\\.[[:digit:]])",  
                        x = r, proto = data.frame(version = character()))
  revision <- strcapture(pattern = "(r[[:digit:]]+)",  x = r,
                         proto = data.frame(revision = character()))
  cbind(version, revision)
}
revision <- data.frame(version = character(),
                       revision = character())
for (i in links) {
  revision <- rbind(revision, extract_revision(i))
}

flavors_df <- cbind(flavors = flavors, flavors_df, revision)
if (!interactive() && !local_build) {
  saveRDS(flavors_df, "flavors.RDS")
}

m <- match(tc$Flavor, flavors_df$flavors)
tc_flavors <- cbind(tc, flavors_df[m, ])
flextable(flavors_df) |> 
  autofit()

flavors	r_version	os	architecture	other	version	revision
r-devel-linux-x86_64-debian-clang	devel	linux	x86_64	debian-clang		r89950
r-devel-linux-x86_64-debian-gcc	devel	linux	x86_64	debian-gcc		r89961
r-devel-linux-x86_64-fedora-clang	devel	linux	x86_64	fedora-clang		r89950
r-devel-linux-x86_64-fedora-gcc	devel	linux	x86_64	fedora-gcc		r89943
r-patched-linux-x86_64	patched	linux	x86_64		4.6.0	r89932
r-release-linux-x86_64	release	linux	x86_64		4.6.0
r-release-macos-arm64	release	macos	arm64		4.6.0	r89937
r-release-macos-x86_64	release	macos	x86_64		4.6.0	r89743
r-release-windows-x86_64	release	windows	x86_64		4.6.0	r89945
r-oldrel-macos-arm64	oldrel	macos	arm64		4.5.2	r89382
r-oldrel-macos-x86_64	oldrel	macos	x86_64		4.5.2	r89382
r-oldrel-windows-x86_64	oldrel	windows	x86_64		4.5.3

It assumes that the same configuration in one package is used for all. Or in other words that the reports of the configuration (svn revision and version) for the A3 package is the same as for all the other packages.

Warning: This assumption is not always true, but this would require to check each log file on each flavor to verify the R and svn id of each package (which could take too much time and resources).

Overview

Briefly an introduction of how much effort goes into checking

library("ggplot2")
theme_set(theme_minimal())
tc |> 
  filter(!is.na(T_install)) |> 
  ggplot() +
  geom_violin(aes(T_install, Flavor)) +
  scale_x_log10() +
  labs(x = "seconds", title = "Time to install", y = element_blank())

Machines (y axis) vs install time (seconds, x axis), violing plot usually around 10 seconds.

Distribution of install time on each machine.

This means that just to install all the packages on the multiple flavors with a single CPU would take 63 days.

tc |> 
  filter(!is.na(T_check)) |> 
  ggplot() +
  geom_violin(aes(T_check, Flavor), trim = FALSE) +
  scale_x_log10() +
  labs(x = "seconds", title = "Time to check", y = element_blank())

Machines (y axis) vs check time (seconds, x axis), violing plot usually around 100 seconds.

Distribution of checking time on each machine.

This means that to check all the packages on the multiple flavors with a single CPU would take 303 days.

tc |> 
  filter(!is.na(T_total)) |> 
  ggplot() +
  geom_violin(aes(T_total, Flavor)) +
  scale_x_log10() +
  labs(x = "seconds", title = "Time to check and install", y = element_blank())

Machines (y axis) vs total time (seconds, x axis), violing plot usually around 100 seconds.

Distribution of total time on each machine.

This means that to install and check all the packages with a single CPU would take 379 days.

I don’t know the computational cost of 266 days of CPU (every day), but a rough calculation of 2.5 cents per hour means 227.48 dollars daily dedicated to this.

tc |> 
  group_by(Package) |> 
  summarize(Versions = n_distinct(Version)) |> 
  ungroup() |> 
  count(Versions, name = "Packages", sort = TRUE) |> 
  flextable() |> 
  autofit()

Versions	Packages
1	23,510
2	249
3	3

This was surprising, but sometimes checks have multiple versions. Probably when a new version is added and the system don’t catch it for a certain machine.

tc |> 
  group_by(Package) |> 
  summarize(Flavors = n_distinct(Flavor)) |> 
  ungroup() |> 
  count(Flavors, name = "Packages", sort = TRUE) |> 
  flextable() |> 
  autofit()

Flavors	Packages
12	23,511
11	162
10	35
8	22
9	14
2	13
5	2
7	2
6	1

Similarly, often packages are only tested on few configurations.

Combining both we can have packages with few configurations that have multiple versions being tested.

tc |> 
  group_by(Package) |> 
  summarize(Versions = as.character(n_distinct(Version)),
            Flavors = n_distinct(Flavor)) |> 
  ungroup() |> 
  count(Flavors, Versions, name = "Packages") |> 
  ggplot() +
  geom_tile(aes(Flavors, Versions, fill = log10(Packages))) +
  scale_x_continuous(expand = expansion())

Flavors of machines and versions of packages

Most packages are just tested one version.

But focusing on those that have just one version of the package being tested, most of the machines have packages either OK or with some notes.

man_colors <- c("OK" = "green", "NOTE" = "darkgreen", 
                "WARNING" = "yellow", "ERROR" = "red", "FAILURE" = "black")
tc |> 
  group_by(Package) |> 
  filter(n_distinct(Version) == 1) |> 
  ungroup() |> 
  group_by(Flavor) |> 
  count(Status, name = "packages") |> 
  mutate(perc = packages/sum(packages),
         Status = forcats::fct_relevel(Status, names(man_colors))) |> 
  ggplot() + 
  geom_col(aes(perc, Flavor, fill = Status)) +
  scale_x_continuous(expand = expansion(), labels = scales::percent_format()) +
  scale_fill_manual(values = man_colors) +
  labs(title = "Packages check status", x = element_blank())

On the vertical axis the machine, on the horitzonal axis the packages colored by the status.

Most frequent status is OK or NOTE on all machines.

If we look at the most frequent status report for packages we can see this table:

ts <- tc |> 
  group_by(Package) |> 
  filter(n_distinct(Version) == 1) |> 
  count(Status, name = "flavors") |> 
  ungroup() |> 
  tidyr::pivot_wider(values_from = flavors, names_from = Status, 
                     values_fill = 0) |> 
  count(OK, NOTE, WARNING, ERROR, FAILURE, name = "packages", sort = TRUE)
download.file("https://cran.r-project.org/web/packages/packages.rds", 
              destfile = "packages.RDS") # From the help page
ap <- readRDS("packages.RDS") |> 
  as.data.frame() |> 
  distinct(Package, .keep_all = TRUE)
ap_bioc <- available.packages(repos = BiocManager::repositories()[1:5])
ap_bioc <- cbind(ap_bioc, Additional_repositories = NA)
ap_colm <- intersect(colnames(ap), colnames(ap_bioc))
ap <- rbind(ap[, ap_colm], ap_bioc[, ap_colm])
head(ts) |> 
  flextable() |> 
  autofit()

OK	NOTE	ERROR	packages
12	0	0	15,929
10	2	0	4,353
0	12	0	1,156
11	0	1	694
11	1	0	292
8	4	0	285

We can see that the most common occurrences are some sort of OK and notes on checks. We can also check the official results on CRAN.

We can see that 57.85% of packages pass all checks without notes.

Now let’s see which of the notes or failures are due to intermittent issues.

Compare

First we need to make sure that we compare the right configurations. They must be the same machine, the same R version and the same svn revision between yesterday and today.

# Compare the previous flavor with today's
m_flavor <- which(flavors_df$flavors %in% old_flavors$flavors)
m_version <- which(flavors_df$version %in% old_flavors$version)
m_revision <- which(flavors_df$revision %in% old_flavors$revision)
tm <- table(c(m_flavor, m_version, m_revision))
compare <- flavors_df$flavors[tm == 3] # Only missing the packages version

All changes

Next, compare the status of the packages if the version of the package is the same.

# Find package on the flavors to compare that haven't changed versions
library("dplyr")
tcc <- filter(tc, Flavor %in% compare) |> 
  select(Flavor, Package, Version, Status) |> 
  arrange(Flavor, Package)
ycc <- filter(yc, Flavor %in% compare) |> 
  select(Flavor, Package, Version, Status) |> 
  arrange(Flavor, Package)

all_checks <- merge(tcc, ycc, by = c("Flavor", "Package"), 
                    suffixes = c(".t", ".y"), all = TRUE) 

possible_packages <- all_checks |> 
  filter(Version.t == Version.y & # Same version
           Status.t != Status.y & # Different status
           !is.na(Status.y) & # No new version or removed package
           !is.na(Status.t)) |> 
  rename(Today = Status.t, Yesterday = Status.y)
possible_packages |> 
  select(Package, Flavor, Today, Yesterday, -Version.t, -Version.y) |> 
  arrange(Package, Flavor) |> 
  flextable() |> 
  autofit()

Package	Flavor	Today	Yesterday
AutoWMM	r-oldrel-windows-x86_64	OK	ERROR
BIEN	r-release-linux-x86_64	OK	ERROR
CAESAR.Suite	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
CongreveLamsdell2016	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
CongreveLamsdell2016	r-release-linux-x86_64	ERROR	OK
DataQualityDashboard	r-release-linux-x86_64	ERROR	OK
DiagrammeR	r-release-linux-x86_64	ERROR	OK
DramaAnalysis	r-release-linux-x86_64	NOTE	OK
EcoCleanR	r-release-linux-x86_64	OK	ERROR
FCPS	r-release-linux-x86_64	NOTE	OK
FastRWeb	r-release-linux-x86_64	OK	NOTE
FracFixR	r-oldrel-windows-x86_64	OK	FAILURE
ISwR	r-release-linux-x86_64	NOTE	OK
JWileymisc	r-release-linux-x86_64	OK	ERROR
LABTNSCPSS	r-oldrel-windows-x86_64	ERROR	OK
PNADCperiods	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
ProFAST	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
ProjectManagement	r-release-linux-x86_64	OK	NOTE
Quartet	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
Quartet	r-release-linux-x86_64	ERROR	OK
R2jags	r-release-linux-x86_64	OK	ERROR
RPesto	r-release-linux-x86_64	NOTE	OK
Rdiagnosislist	r-release-linux-x86_64	NOTE	OK
Rogue	r-devel-linux-x86_64-fedora-gcc	ERROR	NOTE
Rogue	r-release-linux-x86_64	ERROR	OK
SMITIDstruct	r-release-linux-x86_64	NOTE	OK
SQLFormatteR	r-release-linux-x86_64	NOTE	OK
SeqFeatR	r-release-linux-x86_64	NOTE	OK
TBRDist	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
TBRDist	r-release-linux-x86_64	ERROR	OK
Ternary	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
Ternary	r-release-linux-x86_64	ERROR	OK
TreeDist	r-release-linux-x86_64	ERROR	OK
TreeSearch	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
TreeSearch	r-release-linux-x86_64	ERROR	OK
TrueWAP	r-release-linux-x86_64	OK	NOTE
VBTree	r-release-linux-x86_64	NOTE	OK
aggreCAT	r-release-linux-x86_64	NOTE	OK
arcpbf	r-release-linux-x86_64	NOTE	OK
awdb	r-release-linux-x86_64	NOTE	OK
bayesdfa	r-release-linux-x86_64	ERROR	OK
bexy	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
bexy	r-release-linux-x86_64	ERROR	OK
blavaan	r-release-linux-x86_64	ERROR	OK
brainGraph	r-release-linux-x86_64	NOTE	OK
brxx	r-release-linux-x86_64	ERROR	OK
coFAST	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
comstab	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
comstab	r-release-linux-x86_64	ERROR	OK
data.table	r-release-linux-x86_64	ERROR	NOTE
datefixR	r-release-linux-x86_64	NOTE	OK
dplyr	r-release-linux-x86_64	OK	NOTE
easynem	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
easynem	r-release-linux-x86_64	ERROR	OK
efdm	r-release-linux-x86_64	ERROR	OK
eudata	r-release-linux-x86_64	ERROR	OK
fcl	r-release-linux-x86_64	NOTE	OK
fingerPro	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
fingerPro	r-release-linux-x86_64	ERROR	OK
flint	r-release-linux-x86_64	OK	NOTE
fozziejoin	r-release-linux-x86_64	NOTE	OK
fuzzyRankTests	r-release-linux-x86_64	OK	NOTE
ggdibbler	r-release-linux-x86_64	WARNING	OK
ggsem	r-release-linux-x86_64	ERROR	OK
gitGPT	r-release-linux-x86_64	NOTE	OK
gstat	r-release-linux-x86_64	OK	NOTE
heck	r-release-linux-x86_64	NOTE	OK
iotools	r-release-linux-x86_64	OK	NOTE
istacr	r-release-linux-x86_64	ERROR	OK
jewel	r-release-linux-x86_64	ERROR	OK
jrSiCKLSNMF	r-release-linux-x86_64	WARNING	OK
junco	r-release-linux-x86_64	ERROR	OK
lazyeval	r-release-linux-x86_64	OK	NOTE
llamaR	r-release-linux-x86_64	OK	ERROR
llmjson	r-release-linux-x86_64	NOTE	OK
lme4	r-release-linux-x86_64	OK	ERROR
lobstr	r-release-linux-x86_64	OK	NOTE
luajr	r-release-linux-x86_64	NOTE	OK
magrittr	r-release-linux-x86_64	OK	NOTE
marsrad	r-release-linux-x86_64	NOTE	OK
metadat	r-release-linux-x86_64	ERROR	OK
mrIML	r-release-linux-x86_64	OK	ERROR
neo4r	r-release-linux-x86_64	NOTE	OK
orbweaver	r-release-linux-x86_64	NOTE	OK
plot3logit	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
plot3logit	r-release-linux-x86_64	ERROR	OK
polite	r-release-linux-x86_64	NOTE	OK
priceR	r-release-linux-x86_64	NOTE	OK
promor	r-release-linux-x86_64	ERROR	OK
rFIA	r-release-linux-x86_64	WARNING	OK
rGhanaCensus	r-release-linux-x86_64	ERROR	NOTE
rMVP	r-release-linux-x86_64	FAILURE	OK
rSpectral	r-release-linux-x86_64	ERROR	OK
rbm25	r-release-linux-x86_64	NOTE	OK
readmit	r-release-linux-x86_64	ERROR	OK
rgexf	r-release-linux-x86_64	NOTE	OK
rip.opencv	r-release-linux-x86_64	WARNING	OK
rlang	r-release-linux-x86_64	OK	NOTE
rsat	r-release-linux-x86_64	ERROR	OK
rsolr	r-release-linux-x86_64	ERROR	OK
rtiktoken	r-release-linux-x86_64	NOTE	OK
rwty	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
rwty	r-release-linux-x86_64	ERROR	OK
s2	r-release-linux-x86_64	WARNING	OK
scAnnotate	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
sd2R	r-release-linux-x86_64	OK	ERROR
sf	r-release-linux-x86_64	OK	NOTE
sfnetworks	r-release-linux-x86_64	ERROR	OK
sfsmisc	r-release-linux-x86_64	NOTE	OK
sftrack	r-release-linux-x86_64	NOTE	OK
shinyNORRRM	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
shinyNORRRM	r-release-linux-x86_64	ERROR	OK
sparsevctrs	r-release-linux-x86_64	OK	NOTE
spatialrisk	r-release-linux-x86_64	ERROR	OK
stars	r-release-linux-x86_64	OK	NOTE
symphony	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
taylor	r-release-linux-x86_64	OK	ERROR
tidystringdist	r-release-linux-x86_64	NOTE	OK
tmap.networks	r-release-linux-x86_64	ERROR	OK
treesitter	r-release-linux-x86_64	OK	NOTE
trud	r-release-linux-x86_64	OK	ERROR
unsum	r-release-linux-x86_64	NOTE	OK
vctrs	r-release-linux-x86_64	OK	NOTE
vroom	r-release-linux-x86_64	OK	NOTE
waterquality	r-release-linux-x86_64	ERROR	OK
whSample	r-release-linux-x86_64	WARNING	OK
wrProteo	r-release-linux-x86_64	ERROR	OK
xactonomial	r-release-linux-x86_64	NOTE	OK
ymd	r-release-linux-x86_64	NOTE	OK
zoomerjoin	r-release-linux-x86_64	NOTE	OK

If the machine and R versions is the same but the check of the package is different there might be some discrepancy between the dependencies.

# Extract dependencies
dependencies <- package_dependencies(unique(possible_packages$Package),
                                     # Should it check all the recursive dependencies or only direct?
                                     db = ap, # Only considering those dependencies on CRAN and Bioconductor but not any Additional_repositories. 
                                     recursive = TRUE, 
                                     which = c("Depends", "Imports", "LinkingTo", "Suggests"))

# Prepare to compare versions (as they are sorted by everything else we can compare directly)
intermittent_failures <- rep(FALSE, length(dependencies))
names(intermittent_failures) <- names(dependencies)
dep_0 <- lengths(dependencies) == 0
intermittent_failures[dep_0] <- TRUE

If they do not have any recursive dependency on Depends, Imports, LinkingTo and Suggests they might be have some intermittent problems on the packages. These is only on dependencies on CRAN and Bioconductor but not in other additional repositories (There are 196 packages with additional repositories).

If they have some dependencies and those dependencies didn’t change as far as we can tell then there might be some problems with random numbers or connectivity.

for (pkg in names(intermittent_failures[!intermittent_failures])) {
  dep <- dependencies[[pkg]]
  fl <- possible_packages$Flavor[possible_packages$Package == pkg]
  intermittent_failures[pkg] <- all_checks |> 
    filter(Package %in% dep,
           Flavor %in% fl,
           Version.t == Version.y,
           Status.t != Status.y) |> 
    nrow() == 0 # If packages outside || any(!dep %in% rownames(ap)) 
}
packages <- names(intermittent_failures)[intermittent_failures]

We finally show the differences on the status of those without any dependency change on version or status¹:

keep_files <- filter(possible_packages, Package %in% packages) |> 
  merge(y = flavors_df, by.x = "Flavor", by.y = "flavors", all.x = TRUE, all.y = FALSE) |> 
  select(Package, Flavor, Version = Version.t, R_version = r_version, OS = os, 
         architecture, other, version, revision) |> 
  mutate(Date = Sys.time())

if (nrow(keep_files >= 1)) {
  write.csv(keep_files, 
            paste0("cran-failing-", format(Sys.time(), "%Y%m%dT%H%M"), ".csv"),
            row.names = FALSE,
            quote = FALSE,
  )
}

filter(possible_packages, Package %in% packages) |> 
  select(Package, Flavor, Today, Yesterday, -Version.t, -Version.y) |> 
  flextable() |> 
  autofit()

Package	Flavor	Today	Yesterday
PNADCperiods	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
ProFAST	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
scAnnotate	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
symphony	r-devel-linux-x86_64-fedora-gcc	ERROR	OK
AutoWMM	r-oldrel-windows-x86_64	OK	ERROR
FracFixR	r-oldrel-windows-x86_64	OK	FAILURE
LABTNSCPSS	r-oldrel-windows-x86_64	ERROR	OK
FastRWeb	r-release-linux-x86_64	OK	NOTE
flint	r-release-linux-x86_64	OK	NOTE
iotools	r-release-linux-x86_64	OK	NOTE

Conclusion

cat("There are no packages detected with differences between yesterday and today attributable to intermittent failures.\n")
knitr::knit_exit()

cat("This suggests that these packages might have some problems with random numbers or connectivity:\n\n")

This suggests that these packages might have some problems with random numbers or connectivity:

if (any(dep_0)) {
  cat("\n## Packages with dependencies\n\n")
  cat(paste0(" - ", sort(intersect(packages, 
                                   names(dependencies)[dep_0])), "\n"), sep = "")
  cat("\n## Packages without dependencies\n\n")
  cat(paste0(" - ", sort(intersect(packages,
                                   names(dependencies)[!dep_0])), "\n"), sep = "")
  
} else {
  cat(paste0(" - ", sort(packages), "\n"), sep = "")
}

AutoWMM
FastRWeb
flint
FracFixR
iotools
LABTNSCPSS
PNADCperiods
ProFAST
scAnnotate
symphony

I think a new version might not propagate to check other packages until 24 hours later as checks might have already started for that day.↩︎

Packages intermittent failtures

Lluís Revilla Sancho

2026-04-25