Saving a simList may not work using the standard approaches (e.g.,
save, saveRDS, and qs::qsave). There are 2 primary reasons why this doesn't
work as expected: the activeBindings that are in place within modules (these
allow the mod and Par to exist), and file-backed rasters. Because of these,
a user should use saveSimList and loadSimList (and the zipSimList/unzipSimList
alternatives).
The most robust way if there are file-backed Raster* objects seems to be to
set fileBackend = 2, though this may not be desirable if there are many
large Raster* objects. When using fileBackend = 0 or fileBackend = 1, and
when errors are noticed, please file a bug report on GitHub.
zipSimList will save the simList and file-backed Raster* objects, plus,
optionally, files identified in outputs(sim) and inputs(sim).
This uses Copy under the hood, to not affect the original
simList.
VERY experimental.
saveSimList(sim, filename, fileBackend = 0, filebackedDir = NULL, envir, ...)
zipSimList(sim, zipfile, ..., outputs = TRUE, inputs = TRUE, cache = FALSE)Either a simList or a character string of the name
of a simList that can be found in envir. Using
a character string will assign that object name to the saved
simList, so when it is recovered it will be given that
name.
Character string with the path for saving simList to or
reading the simList from. Currently, only .rds and .qs file types are supported.
Numeric. 0 means don't do anything with
file backed rasters. Leave their file intact as is, in its place.
1 means save a copy of the file backed rasters in fileBackedDir.
2 means move all data in file-backed rasters to memory. This
means that the objects will be part of the main qs file
of the simList. Default is 0.
Only used if fileBackend is 1.
NULL, the default, or Character string. If NULL, then then the
files will be copied to the directory:
file.path(dirname(filename), "rasters"). A character string
will be interpreted as a path to copy all rasters to.
If sim is a character string, then this must be provided.
It is the environment where the object named sim can be found.
passed to saveSimList(), including non-optional ones
such as filename. Also see fileBackend and filebackedDir
arguments in that function.
A character string indicating the filename for the zip file. Passed to zip.
Logical. If TRUE, all files identified in
outputs(sim) will be included in the zip.
Logical. If TRUE, all files identified in
inputs(sim) will be included in the zip.
Logical. Not yet implemented. If TRUE, all files in cachePath(sim) will be included in the
zip archive. Defaults to FALSE as this could be large, and may include many
out of date elements. See Details.
saveSimList(): A saved .qs file in filename location.
zipSimList(): A saved .zip file in zipfile location.
There is a family of 4 functions that are mutually useful for saving and
loading simList objects and their associated files (e.g., file-backed
Raster*, inputs, outputs, cache) saveSimList(), loadSimList(),
zipSimList(), unzipSimList()
There are 3 different workflows for "save - move files - load" that work in our tests:
filebackend = 0: No renaming of file-backed rasters, on recovery attempts to rebuild
This approach is attempting to emulate a "relative filenames" approach,
i.e., attempt to treat the file-backed raster file names as if they were
relative (which they are not -- raster package forces absolute file
paths). To do this, all the renaming occurs within loadSimList or
unzipSimList. These function will use the paths argument to rewrite
the paths of the files that are identified with Filenames(sim) so that
they are in the equivalent (relative) position as they were. This will
only work if all files were in one of the paths of the original
simList, so that they can be matched up with the new paths passed in
loadSimList. This is not guaranteed to work correctly, though it works
in a wide array of testing.
zipSimList(sim, zipfile = tmpZip, filename = "sim.qs")
pths <- getPaths(mySim)
out <- unzipSimList(tmpZip, paths = pths)filebackend = 1: On-the-fly renaming of file-backed rasters;
Save the sim object with a filename, e.g., file,
make a copy of all file-backed rasters to fileBackedDir,
update all the pointers to those files so that they are correct in the raster metadata
saveSimList(sim, file = "sim.qs", fileBackend = 1, fileBackedDir = "here")
simNew <- loadSimList(file = "sim.qs")filebackend = 2: On-the-fly bringing to memory of all rasters
All rasters are brought to memory, and then saved into sim.qs
saveSimList(sim, file = "sim.qs", fileBackend = 2)
simNew <- loadSimList(file = "sim.qs")If cache is used, it is likely that it should be trimmed before
zipping, to include only cache elements that are relevant.