Combine Results Saved by save_objects with incremental=TRUE

This function is only intended to be used after save_objects with incremental=TRUE. In this case, save_objects with save results under temporary hashes in a folder with the hash corresponding the the parameters. compress_incremental then combines the results and saves them under the corresponding hash and deletes the old directory with the temporary results.

Usage

compress_incremental(
  folder,
  parameters_list,
  hash_includes_timestamp = FALSE,
  ignore_na = TRUE,
  alphabetical_order = TRUE,
  algo = "xxhash64",
  ignore_script_name = FALSE,
  remove_folder = TRUE
)

Arguments

folder

Character string specifying the path to the directory where the temporary folder was saved (should be the same as supplied to save_objects.

parameters_list

The named list of arguments used with save_objects.

hash_includes_timestamp

Logical. If TRUE, the timestamp is included in the hash generation.

ignore_na

Logical. If TRUE, NA values in parameters_list are ignored during hash generation.

alphabetical_order

Logical. If TRUE, the names in parameters_list are sorted alphabetically before hash generation.

algo

Character string specifying the hashing algorithm to use. Default is "xxhash64". See ?digest

ignore_script_name

Logical. If TRUE, the script name is ignored during hash generation.

remove_folder

Logical. If TRUE, the folder and the temporary results files will be discarded after the combined results are saved.

Value

No return value. This function is called for its side effects.

Details

If the individual results can be put into a data.frame they will be, otherwise they will be stored as a list.

See also

save_objects()

Examples

## Save results incrementally
params <- list(a = "1", b = "2")

tmp_dir <- file.path(tempdir(), "example")
dir.create(tmp_dir)
for (i in 1:10) {
  save_objects(tmp_dir, data.frame(idx = i, val = rnorm(1)), params, incremental = TRUE)
}

## See contents of tmp directory for incremental file
list.files(file.path(tmp_dir, generate_hash(params)))
#> character(0)

## Compress results into a single file
compress_incremental(tmp_dir, params)
#> Error in compress_incremental(tmp_dir, params): Incremental folder does not exist: /tmp/Rtmp9Prjgj/example/1ca71bd5b88c116d
list.files(tmp_dir)
#> [1] "67159522d5241f97"

## Read in compressed file and view results
read_objects(tmp_dir, params)
#> Warning: File not found for hash: 1ca71bd5b88c116d
#> NULL

## Cleanup
unlink(tmp_dir, recursive = TRUE)