What Is Memory-Mapping?
Memory-mapping is a mechanism that maps a portion of a file,or an entire file, on disk to a range of addresses within an application'saddress space. The application can then access files on disk in thesame way it accesses dynamic memory. This makes file reads and writesfaster in comparison with using functions such as fread and fwrite.
Benefits of Memory-Mapping
The principal benefits of memory-mapping are efficiency, fasterfile access, the ability to share memory between applications, andmore efficient coding.
Faster File Access
Accessing files via memory map is faster than using I/O functionssuch as fread and fwrite. Data are read and written usingthe virtual memory capabilities that are built in to the operatingsystem rather than having to allocate, copy into, and then deallocatedata buffers owned by the process.
MATLAB® does not access data from the disk when the mapis first constructed. It only reads or writes the file on disk whena specified part of the memory map is accessed, and then it only readsthat specific part. This provides faster random access to the mappeddata.
Efficiency
Mapping a file into memory allows access to data in the fileas if that data had been read into an array in the application's addressspace. Initially, MATLAB only allocates address space for thearray; it does not actually read data from the file until you accessthe mapped region. As a result, memory-mapped files provide a mechanismby which applications can access data segments in an extremely largefile without having to read the entire file into memory first.
Efficient Coding Style
Memory-mapping in your MATLAB application enables you toaccess file data using standard MATLAB indexing operations. Onceyou have mapped a file to memory, you can read the contents of thatfile using the same type of MATLAB statements used to read variablesfrom the MATLAB workspace. The contents of the mapped file appearas if they were an array in the currently active workspace. You simplyindex into this array to read or write the desired data from the file.Therefore, you do not need explicit calls to the fread and fwrite functions.
In MATLAB, if x is a memory-mapped variable,and y is the data to be written to a file, thenwriting to the file is as simple as
x.Data = y;
Sharing Memory Between Applications
Memory-mapped files also provide a mechanism for sharing databetween applications, as shown in the figure below. This is achievedby having each application map sections of the same file. You canuse this feature to transfer large data sets between MATLAB andother applications.
Also, within a single application, you can map the same segmentof a file more than once.
When to Use Memory-Mapping
Just how much advantage you get from mapping a file to memorydepends mostly on the size and format of the file, the way in whichdata in the file is used, and the computer platform you are using.
When Memory-Mapping Is Most Useful
Memory-mapping works best with binary files, and in the followingscenarios:
For large files that you want to access randomly oneor more times
For small files that you want to read into memoryonce and access frequently
For data that you want to share between applications
When you want to work with data in a file as if itwere a MATLAB array
When the Advantage Is Less Significant
The following types of files do not fully use the benefits ofmemory-mapping:
Formatted binary files like HDF or TIFF that requirecustomized readers are not good for memory-mapping. Describing thedata contained in these files can be a very complex task. Also, youcannot access data directly from the mapped segment, but must insteadcreate arrays to hold the data.
Text or ASCII files require that you convert the textin the mapped region to an appropriate type for the data to be meaningful.This takes up additional address space.
Files that are larger than several hundred megabytesin size consume a significant amount of the virtual address spaceneeded by MATLAB to process your program. Mapping files of thissize may result in MATLAB reporting out-of-memory errors moreoften. This is more likely if MATLAB has been running for sometime, or if the memory used by MATLAB becomes fragmented.
Maximum Size of a Memory Map
Due to limits set by the operating system and MATLAB, themaximum amount of data you can map with a single instance of a memorymap is 2 gigabytes on 32-bit systems, and 256 terabytes on 64-bitsystems. If you need to map more than this limit, you can either createseparate maps for different regions of the file, or you can move thewindow of one map to different locations in the file.
Byte Ordering
Memory-mapping works only with data that have the same byteordering scheme as the native byte ordering of your operating system.For example, because both Linus Torvalds' Linux® and Microsoft® Windows® systemsuse little-endian byte ordering, data created on a Linux systemcan be read on Windows systems. You can use the computer function to determine the nativebyte ordering of your current system.
