NEWS

filearray 0.1.9 (2024-11-08)

Fixed CRAN check issue brought by testthat update, which requires bit64 when checking int64 arrays. Added bit64 to Suggests field to mitigate this check error

Fixed a bug where C++ errors are not correctly thrown due to a recent Rcpp update. This bug may cause erroneous code to execute ignoring try-catch statement and imbalanced protecting stacks (fwhich, mapreduce)

Removed non-API R C interface
Allows subsetAssign to assign when value length is 1 (thanks @talegari issue #9)
Allows more flexible fwhich function with functional val input and options to include corresponding values as part of return (thanks @talegari issue #10)
Fixed mapreduce bug when partition length is way larger than the buffer size, causing buffer to stop updating
Fixed other C++ issues, such as unused buffer allocation in subset-assign, and min_ offset in sub_vec_range. These issues are not considered bugs since they don't affect the results, but the performance is slightly increased after fix.
Replaced dipsaus with more updated ravetools in the vignette performance profiling
Updated examples to get rid of warning messages and out-dated arguments.

Removed c++11 from system requirement and Makevars
Used TinyThreads instead of OpenMP to get parallel working on OSX
Added as_filearray method, and support optional float
Added array proxy class, allowing arrays to lazy-evaluate simple operators
Allows user-defined temporary file array path
Fixed fmap issues, using better guesses for default .input_size
Fixed a memory bug caused when partition margin has elements greater than 1 and when FARR_subset_sequential is used
Used fastmap to avoid environment look-up
fa_eval_ops allows dimension names
Larger default buffer size to allow 2^20 in single partition
Disabled single indexing
Fixed endian issue on big-endian platforms

Fixed a bug when trying to read array data sequentially. The bug is caused by buffer size being greater than the array length, making in a pointer that controls the partition number exceed the end of vector, resulting in undefined behavior. The functions affected are: fmap, fmap2. The bug has been fixed and passed valgrind memory check.

Fixed a bug when allocated memory is one byte short than requested. The bug would crash R when triggered in certain cases.
Removed limit to the maximum number of partitions when writing. The previous implementation creates and opens related file descriptors all at once before writing. This setup will raise errors when the number of connections reach to certain limit, often defined by the operating systems. This update only opens the connection on demand. The performance might be impacted when writing to disk, but in return, the program will be more robust
Fixed subset function environment not resolved correctly when using formula
Added filearray_load_or_create as an alternative to filearray_checkload by automatically replace existing obsolete array files if the headers, dimensions, or data types don't match. Also on_missing argument is provided to allow array initialization if new array is created.

Added OpenMP flag in the MakeVars
Fixed critical bugs that could cause segfaults
Can store complex and float data types
Re-implemented read/write functions to use memory map
Allowed dimnames to be set
Added generics subset to subset using dimnames
Added vignette to compare performance
Added speed comparisons in README.md
Added collapse to calculate marginal summation with little memory overhead
Added fmap, fmap2 to apply functions to one or multiple file arrays with little memory overhead (also very fast)
Fixed 'unprotected' issues warned by rchk