Host File System vs. Hardfiles
TOPIC
Q: Is it better to store Amiga files using the host file system or using hardfiles? What about using a real Amiga hard disk instead of a hardfile?
DISCUSSION
A: Using the host file system (e.g. Windows) means that each Amiga file is handled by the file system used by the operating system on which the emulation is running (e.g. NTFS, on Windows). The host file system is seen by the Amiga as if it were an Amiga-compatible file system (e.g. like a CD or network file system), with very minor differences involving file names and attributes compared to the original Amiga OFS and FFS file systems.
A hardfile (also called HDF file) is similar to a hard disk partition, but is stored as a single file on the host file system. Data handling inside the hardfile (i.e. the actual reading and writing of Amiga files and directories) is managed by the Amiga operating system running in the emulation, not by the host operating system.
Both methods have advantages and disadvantages. If the content is to be embedded in an RP9 package, we recommend using HDF files. Embedding of large content (e.g. 100 MB or more) inside an RP9 package can slow down the initial startup and the final shutdown (to commit changes), because of the decompression and compression overhead (RP9 packages are compressed), so it may be best to configure larger hardfiles as external references (opposed to embedded files) in the RP9 Title Editor.
Amiga Forever also supports "deployable" hardfiles, which combine the portability benefits of embedded disks with the superior performance of external images. These are recommended for RP9 packages that contain large disk images, but which also need to be moved between systems.
If the content is to be shared outside of RP9 packages we recommend the use of the host file system for everyday work, and to use hardfiles only if you need to store Amiga data where it is important to exactly preserve the original file attributes (exact date, Amiga flags, comment, special characters in name, etc.) While hardfiles have some interesting advantages for archival purposes, the disadvantages tend to prevail when the operational aspect is considered. At Cloanto we use hardfiles for the archival of our original Amiga projects and disk masters, and the host file system for everything else.
By default, Amiga Forever uses the host file system, however hardfiles can be created and mounted manually. The Amiga Forever hard disk installation creates a "Work" directory on the PC hard disk which is seen as a hard disk named "Work" from within the emulation environment. This directory is located in a path like "Shared/dir/Work", inside "Amiga Files" (in Amiga Forever, select Open Folder/Amiga Files from the File menu). By copying files from the Amiga hard disk to this directory on the PC, they automatically become available in the "Work" partition of the emulated Amiga. The same is done for the "System" directory, which contains the Amiga system files. Inside the Amiga Workbench, remember to set the Show/All files option to view files that don't have an associated .info icon file.
Advantages of using the host file system include:
- Data can be accessed by both the Amiga and the Windows (or other operating system) side
- Disk usage is always optimal (no risk to reach the end of a hardfile, or to waste unused space)
- NTFS file compression, if enabled, can be more efficient and faster on single files than on a larger hardfile
- The host file system usually offers more features (compression, encryption, atomic transactions, etc.) and is designed to support newer storage devices (e.g. Blu-ray, flash memory, etc.) and hard disks with terabyte-class capacities
Disadvantages of using the host file system (see below for more details):
- Certain characters (e.g. "\") and names (e.g. "AUX" and single space character) which are supported in Amiga file and directory names may be illegal or have a special meaning on the host file system
- Amiga-specific attribute bits (e.g. "script", "pure", etc.) and comments may not be supported
- The original Amiga file and directory timestamps may be slightly altered (by small fractions of a second) due to the different timing granularity of the host file system
- Directory structure cannot be embedded as is in RP9 packages (but can be added as external references)
- When used under a PowerPC emulation, a second layer of 68K emulation is required to support file system features of mounted directories, which reduces performance
Advantages of hardfiles:
- Supports all Amiga file systems (e.g. OFS, FFS), including third party ones (SFS, PFS, etc.)
- Ideal for one-time archiving of Amiga hard disk partitions (e.g. using Amiga Explorer)
- Can be embedded in RP9 packages
Disadvantages of hardfiles:
- Data corruption and/or medium defects may affect an entire hardfile (not just the most recently used data file) and/or may require the use of recovery tools on both the Amiga side and the host operating system side, making recovery attempts significantly more complex
- Similar to a hard disk partition, a hardfile's allocated size is constant, and cannot (easily) shrink or grow as needs change
- The content of a hardfile cannot (usually) be accessed by other operating systems (e.g. Windows)
Some versions of the Amiga emulation can directly access a real Amiga-formatted hard disk. The considerations already outlined for hardfiles usually apply to this case, with the added consideration that an Amiga hard drive may have a certain age, and it may therefore be even more prudent to copy the data to a newer hard disk (e.g. to the host file system, so it can be shared by the Amiga and the other operating system).
The Amiga file system, like the older Windows file systems, was created when disks of a few dozen megabytes (not today's gigabytes and terabytes) were considered a luxury, and were not designed to support drives introduced several decades later. If a software error or power failure or "dirty" shutdown occurs during a write operation, it is not uncommon for a "repair" operation to take hours or days, as it may have to scan hundreds of gigabytes of data, if not more. Newer PC file systems, such as HPFS, NTFS and ReFS, are based on atomic transactions. A transaction can't be "half-complete": if a write operation fails the transaction remains logged as incomplete, and the overall file system structure remains in a known state, without corrupt pointers or links. Time consuming disk "repair" or "validate" procedures at startup are therefore usually not required. Using the host file system in the emulation, rather than a hard file or a real Amiga hard drive, provides this inherited advantage.
While it is true that the host file system may not feature the exact set of Amiga file attributes and supported characters, this is not a new limitation for the Amiga. Amiga software developers always had to be open minded about their software being used on a different file system. Amiga computers accessing data on non-Amiga file systems (e.g. an ISO 9660-2 CD-ROM or a Novell network, or an NFS share) were already quite common among Amiga developers in the late 1980s, and they were mentioned by Commodore-Amiga developer notes. The Amiga operating system itself was organized using common sense file names which work well on most file systems, not just the original Amiga file systems (with the exception of the "AUX" and "AUX.info" files, which are of little or no practical use in the emulation, but which may nevertheless not easily be written to and/or deleted on some Windows file systems). Similarly, it has always been good practice not to count on the fact that Amiga script files have the "s" flag set. Amiga script files which are executed implicitly, i.e. without the "Execute" command, will fail when run from a CD-ROM, or from an emulation environment where Amiga-specific file attributes are not supported. This can easily be solved by prefixing the command with an AmigaDOS "Execute" instruction.
On the Amiga, only two characters are "illegal" in file and directory names: ":" and "/". Under Windows, these are: "\", "/", ":", "*", "?", """ (double quote character), "<", ">" and "|". Additionally, both the Amiga and Windows have other characters and file names which can create difficulties in certain environments, e.g. the Amiga Shell, or the Windows Desktop, but these are not considered illegal at the file system level, and can be accessed by the software. In Windows, such file and path names may include: "CON", "PRN", "AUX", "COMx", "LPTx", "$MFT", "CLOCK$" and "NUL" and and all variants with a "dot suffix". File names entirely consisting of dots (".", "..", etc.) are also "problem files". Usually, it is not possible to manipulate these files from the Windows Desktop or command line, but for example the Amiga emulation environment and the Windows NT POSIX shell can access them. This explains why, for example, the "AUX" and "AUX.info" files, used on some Amiga systems, can be seen, but not deleted from the standard Windows Desktop. This is especially annoying when moving entire directories containing these files, since the move process terminates with an error. If it is not possible to avoid the use of such file names, the best option is usually to store them on an Amiga hardfile rather than on a mounted directory.
Amiga Explorer can be used to copy files, directories or partitions from an Amiga to the PC. Amiga Explorer also comes with Amiga Forever Plus Edition and Premium Edition, or it can be downloaded and registered using the same Amiga Forever license key.
When copying files and directories, "illegal" names are automatically converted by Amiga Explorer during the copy process.
Amiga Explorer also gives access to a "virtual hardfile" for each Amiga hard disk. This is in HDF format, which can be copied directly to the emulation. To mount such an HDF file, view its Properties with Amiga Explorer, and after copying the file to the PC, set the drive geometry values (indicated by Amiga Explorer) in the Amiga Forever configuration (right-click the title, select Edit, then Media tab). We recommend to take a note of these important properties when the hardfile is copied, and to save them in a text file stored in the same directory as the hardfile.
Amiga Forever and Amiga Explorer were created by Amiga enthusiasts who, like so many other Amiga users, wanted to preserve years of hard work. In this spirit, when copying important hard disk data from the Amiga to the PC, we recommend to copy the data twice, i.e. once using hardfiles, and once using the host file system (i.e. dragging and dropping the relevant files and directories from the Amiga to the PC). If possible, the data should be verified with a tool such as DirDiff (with the "Quick" option disabled), which is included on Amiga Forever. We then advise to store a copy of this data on a "safe" medium like a DVD, and to store it in a dry room which is not exposed to day/night temperature fluctuations (which are known to reduce the shelf life of DVDs and other media), preferably not in the same building as the computer. For long-term storage, DVD have a better track record than CDs because the data layer is protected better. For both CDs and DVDs it is important not to attach labels to the medium (on CDs they may rip off the data layer itself, which is just under the surface, while on both CDs and DVDs they may create tensions over time, and vibrations when spinning).
ADF files (and variations such as ADZ, FDI and DMS) are the floppy-disk equivalent of hardfiles. Like that of a hard disk partition, the content of an Amiga floppy disk can be copied and used in the emulation either using the host file system, or mounting the floppy disk image. If the file system is used, and an Amiga with hard disk is available, it is usually faster to first copy the data from the floppy disks to the Amiga hard drive, organizing the data in the process, and then copy the entire set from the Amiga hard disk to the PC.
Related Links
- Amiga SCSI and IDE Devices: Disk Image Files and Direct Access
- Referencing External Content in RP9 Files
- "Deployable" RP9s
- Transferring Software from Amiga to PC
- Reading Amiga Non-Floppy Media on PC
- Reading Amiga Floppy Disks on PC
- System Configuration Import Checklist
- The Migration Tips tutorial
- Data Sharing
ARTICLE INFORMATION
Article ID: | 13-156 |
---|---|
Platform: | All |
Products: | Amiga Forever, Amiga Explorer |
Additional Keywords: | backup, winuae |
Last Update: | 2017-06-23 |
It is safe to link to this page.