Save a whole simList object to disk

Saving a simList may not work using the standard approaches (e.g., save, saveRDS, and qs::qsave). There are 2 primary reasons why this doesn't work as expected: the activeBindings that are in place within modules (these allow the mod and Par to exist), and file-backed rasters. Because of these, a user should use saveSimList and loadSimList (and the zipSimList/unzipSimList alternatives). The most robust way if there are file-backed Raster* objects seems to be to set fileBackend = 2, though this may not be desirable if there are many large Raster* objects. When using fileBackend = 0 or fileBackend = 1, and when errors are noticed, please file a bug report on GitHub.

zipSimList will save the simList and file-backed Raster* objects, plus, optionally, files identified in outputs(sim) and inputs(sim). This uses Copy under the hood, to not affect the original simList. VERY experimental.

saveSimList(sim, filename, fileBackend = 0, filebackedDir = NULL, envir, ...)

zipSimList(sim, zipfile, ..., outputs = TRUE, inputs = TRUE, cache = FALSE)

Arguments

sim	Either a simList or a character string of the name of a simList that can be found in envir. Using a character string will assign that object name to the saved simList, so when it is recovered it will be given that name.
filename	Character string with the path for saving simList to or reading the simList from
fileBackend	Numeric. 0 means don't do anything with file backed rasters. Leave their file intact as is, in its place. 1 means save a copy of the file backed rasters in fileBackedDir. 2 means move all data in file-backed rasters to memory. This means that the objects will be part of the main qs file of the simList. Default is 0.
filebackedDir	Only used if fileBackend is 1. NULL, the default, or Character string. If NULL, then then the files will be copied to the directory: file.path(dirname(filename), "rasters"). A character string will be interpreted as a path to copy all rasters to.
envir	If sim is a character string, then this must be provided. It is the environment where the object named sim can be found.
...	passed to saveSimList, including non-optional ones such as filename. Also see fileBackend and filebackedDir arguments in that function.
zipfile	A character string indicating the filename for the zip file. Passed to zip.
outputs	Logical. If TRUE, all files identified in outputs(sim) will be included in the zip.
inputs	Logical. If TRUE, all files identified in inputs(sim) will be included in the zip.
cache	Logical. Not yet implemented. If TRUE, all files in cachePath(sim) will be included in the zip archive. Defaults to FALSE as this could be large, and may include many out of date elements. See Details.

Value

saveSimList(): A saved .qs file in filename location.

zipSimList(): A saved .zip file in zipfile location.

Details

There is a family of 4 functions that are mutually useful for saving and loading simList objects and their associated files (e.g., file-backed Raster*, inputs, outputs, cache) saveSimList(), loadSimList(), zipSimList(), unzipSimList()

Save - Move - Load

There are 3 different workflows for "save - move files - load" that work in our tests:

1. filebackend = 0: No renaming of file-backed rasters, on recovery attempts to rebuild

   This approach is attempting to emulate a "relative filenames" approach, i.e., attempt to treat the file-backed raster file names as if they were relative (which they are not -- raster package forces absolute file paths). To do this, all the renaming occurs within loadSimList or unzipSimList. These function will use the paths argument to rewrite the paths of the files that are identified with Filenames(sim) so that they are in the equivalent (relative) position as they were. This will only work if all files were in one of the paths of the original simList, so that they can be matched up with the new paths passed in loadSimList. This is not guaranteed to work correctly, though it works in a wide array of testing.

   zipSimList(sim, zipfile = tmpZip, filename = "sim.qs")
   pths <- getPaths(mySim)
   out <- unzipSimList(tmpZip, paths = pths)
   

2. filebackend = 1: On the fly renaming of file-backed rasters;

   1. Save the sim object with a filename, e.g., file,

   2. make a copy of all file-backed rasters to fileBackedDir,

   3. update all the pointers to those files so that they are correct in the raster metadata

   saveSimList(sim, file = "sim.qs", fileBackend = 1, fileBackedDir = "here")
   simNew <- loadSimList(file = "sim.qs")
   

3. filebackend = 2: On the fly bringing to memory of all rasters

   All rasters are brought to memory, and then saved into sim.qs

   saveSimList(sim, file = "sim.qs", fileBackend = 2)
   simNew <- loadSimList(file = "sim.qs")
   

If cache is used, it is likely that it should be trimmed before zipping, to include only cache elements that are relevant.

See also

loadSimList(), unzipSimList()