J&P Group J&P BMCG J&P InfoSol J&P Multimedia Labs J&P NoBoS
friends
fun

J&P Group > J&P InfoSol > File Systems

File Systems

 

Introduction <top>

http://www.google.com/search?q=windows+nt+fs+macosx

 

The Best File System? <top>

In general, which is the best file system (FS) depend on who you are and what your needs are. For example:

  • If you are a common end-user [of computers], the best file system is the one - or ones* - you already got as part of operating system, no fuss.
    *) All modern operating systems (Mac, Linux, MS Win) support multiple filesystems straight out of the box. In general with one primary FS and then a number of secondary FSs that are either read/write- or read-only - suppported.
  • For USB drives, in many cases the old FAT32 is a good choice. (Good compatibility with all operating systems, good for moving between different systems).
    One disadvantage with FAT32 on larger drives is it wastes a lot of space, due to it's technical details. (E.g. a file that occupies 4kB on a 500MB Mac HFS+ formatted drive takes 32kB - 8 times as much - on a 160MB FAT32 formatted drive.)
    (E.g. On Mac, NTFS is supported read-only but not read-write. More technical details below in Features of Common FSs.)
  • Advanced users or specific applications (high-end/speed storage for share use, etc) have a multitude of file systems to choose from (see wikipedia for one list).

It all depend on you needs.

Characteristics to consider include:

  • Reliability - the core feature! An unreliable file system is useless (and why it can take 5 to 7 years - one figure - to develop and get a mature file system)
  • Speed
  • User-friendliness, Easy of use, flexibility - (e.g.) length of directory entries (filenames, directories), what characters are allowed
  • Efficiency - don't want to waste a lot of space for smaller files due to technicalities (like FAT on larger drives)

http://www.google.com/search?q=the+best+file+system+ever

http://www.64studio.com/node/362

http://serverfault.com/questions/14235/best-file-system-for-media-server

 

Overview File Systems for Mac, Linux, Windows <top>

Apple Macintosh, Linux/UNIX, Microsoft Windows, plus a few other of interest - when released and brief notes.

A brief overview based on when released (as indication of how modern):

 

1977

 1984

1985

1987

 1992

1993

 1994

1996

1998

1999

2001

2002

2005

Linux

 

  

 

 

  

ext2

  

 

 

ext3

 

 

 

Mac OS

 

 MFS
(OS 1.0)

HFS
(OS 2.1)

 

  

 

  

 

HFS Plus
(OS 8.1)

 

 

HFSJ
(10.2.2)

 

MS PC

FAT12

  

 

FAT16
(DOS 3.31)

 

NTFS v1
(Win NT)

 

FAT32
(Win 95)

 

 

NTFS v5.1
(XP)

 

 

Other

 

  

 

 

 MVFS
(1992?)

 

 XFS
(SG's
IRIX)

 

 

 

 

 

ZFS

More details:

Platform Name Introduced Original OS Notes
Macintosh: MFS 1984 Mac OS 1.0 The first file system for the Macintosh product line from Apple. Macintosh_File_System [W]
HFS 1985 Mac OS 2.1 (31 characters, file names) Hierachical File System [W]
HFS+ 1988 Mac OS 8.1 HFS Plus, also known as Mac OS Extended; larger files, Unicode HFS Plus [W]
HFSJ 2002 Mac OS 10.2.2 HFS+ with journaling. HFSJ [Wikipedia; same as HFS+, Mar'09]
HFSX 2003 Mac OS 10.3 As HFS+ wut w/o HFS wrapper and support case sensitivity
(HFS...)   Mac OS 10.4 Adding support for access control lists to HFS+ (/HFSJ/HFSX).
(ZFS) (no...2009) (! Mac OS 10.6) Were rumors that 10.6 would include ZFS but in the end didn't. See also our OS X 10.6 section.
Linux ext2 1993 (Jan) Linux  
ext3 2001 (Nov) Kernel 2.4.15 the journaled form of ext2
ext4 2008 (Oct) Kernel 2.6.28 Support for larger volumes (1 exabyte), files up to 16 terabytes, and more.
Microsoft: FAT16 1987 MS-DOS 3.31  
NTFS 1993 Windows NT The Windows NT File System.
FAT32 1996 Windows 98 (95b) Larger volumes and files (cmp to FAT16), other technical improvements.
(WinFS) (2003) (TBD) First introduced in 2003, was to be part of MS Vista (2006, 2007) but then pulled and now (still Mar 2009) unclear if/when will be part of any OS from MS. wikipedia WinFS.
Other MVFS 1992~?s (ClearCase) ClearCase's novelty was in its versioned file system (called MVFS: MultiVersion File System)" from en.wikipedia.org/wiki/Rational_ClearCase.
NFS 1984   NFS v1 in RFC1094.
UDF 1995 ISO/ECMA/OSTA Primary use for optical disks, a better alternative to ISO 9660.
(UFS) (way back) (Unix flavors) Collective name for many different versions and variants used in UNIX-based OS'; Mac OS X has one version. en.wikipedia.org/wiki/Unix_File_System.
XFS 1994 Silicon Graphics IRIX One of the oldest journaling file systems, created for SG's high-end workstations, servers
ZFS 2004 Solaris Zettabyte_File_System

 

Features of Common FSs

 

Features of Common FSs <top>

  Macintosh   Linux   MS Windows  
  Mac OS Extended ext2 ext3 FAT32 NTFS  
Introduced HFS+          
  1988   1999   1993  
File names 255 UTF-16 chars 255 255 8.3 (255 w/ LFNs) 254+'.'  
             
Features            
Hard links Partial yes yes no yes  
Soft links Yes yes yes no partial  
Block Journaling No no yes no no  
Meta-data journaling only Yes (HFSJ) no yes no yes  
Case sensitive Partial yes yes no yes  
Case preserving Yes yes yes partial yes  
File change log Yes no no no yes  
Incremental snapshots No no no no yes  
XIP No yes ? no ?  
File system encryption No no no no yes  
Sharing/Cross-Usage            
Mac OS X native ? ? read-write read-only  
Linux - distros in general no yes yes read-write read-write  
Windows XP no no(?) no(?) read-write native  

LFN = Long File Names

 

Technologies

 

Alternate Data Streams, Forks, ...

(This section added Monday 2009-11-16, *after* creating a report on Alternate Data Streams (ADSs). One note: It was in the MFS (Mac) file system Apple introduced resource forks, not in HFS which came a year later.)

Platform File System(s) Technology First
Introd.		  
Apple Macs HFSJ (10.2.2),
HFS Plus (OS 8.1)
HFS (OS 2.1)
MFS (OS 1.0)
Forked files with
Data Fork and
Resource Fork		 1984

"MFS was notable both for introducing resource forks to allow storage of structured data, and for storing metadata needed to support the graphical user interface of Mac OS." [MFS]

  • The Data Fork is the primary location for data storage (executable code, documents contents, etc)
    • All data is correctly moved when copied to other file systems, sent via networks, emails, ...
  • The Resource Fork the additional location and is commonly used for localization information but can be used for anything really.
    • Special precautions needs to be considered when moving to other file systems, systems, than Mac or information will be lost.
MS Win PCs NTFS
(Win NT)		 Alternate Data
Stream (ADS)		 1993

Alternate data stream, a filesystem fork in Microsoft NTFS. Alternate data stream, wikipedia redirects to Fork ....

Same as with the Resource Fork on Mac, special precautions must be considered when moving files with ADS(s) onto other file systems [than NTFS] or information will be lost. This include other common MS Windows file systems like variants of FAT (FAT32, FAT16, ...).

 

 

Journaling File System

A journaling file system is a file system that logs changes to a journal (usually a circular log in a dedicated area) before committing them to the main file system. Such file systems are less likely to become corrupted in the event of power failure or system crash. [wikipedia]

 

Versioning File System

A versioning file system is any computer file system which allows a computer file to exist in several versions at the same time. Thus it is a form of revision control. Most common versioning file systems keep a number of old copies of the file. Some limit the number of changes per minute or per hour to avoid storing large numbers of trivial changes. Others instead take periodic snapshots whose contents can be accessed with similar semantics to normal file access. [wikipedia]

 

IBM Rational ClearCase

ClearCase' with it's MultiVersion File System (MVFS) is an absolute favorite when it come to file systems! The ability to (on *NIX/Linux) create views with all sorts of selection criterias (date - changed before; changed after, tags, owners, ...) deciding which files and versions to view on the underlying file system. And that ALL applications immediately sees only the selected files make this the best file system for anyone serious about development or revision-handling any types of files. Sources, documents, used applications (yes, you can also version-handling application to make sure you can exactly recreate the development environment you used five years ago.

"A distinguishing feature of ClearCase is a proprietary networked filesystem (MVFS: MultiVersion File System) [7] , which can be used to mount VOBs as a virtual file system through a dynamic view, selecting a consistent set of versions and allowing for the production of derived objects. The dynamic view allows this to map to a Software Configuration. This was a departure from the repository/sandbox model, allowing for the early management of artifacts (before they are being checked in, and not limited to these first order configuration items)." [wikipedia]

Cons include:

  • Cost - not everyone can justify $4,600 per user (suggested retail price, July 2009; src) for buying Rational ClearCase.
    • Plus not to forget maintenance- and support contract (don't know off the hand costs, but for these suites, yearly cost can be 10-20% of original purchase price)
  • Administration - installing, setting up, and maintenance such as backing up VOB(s) (Versioned Object Bases), etc.
  • Users learning - this is a very capable solution and you must count with some education.

Additional information:

Supported platforms (from www-01.ibm.com/support/docview.wss?uid=swg27014067, July 2009; include more details):

Table 1. Supported file systems by platform
Platform Supported file systems
HP-UX JFS, UFS, HFS, VxFS
HP-UX/IPF JFS, UFS, HFS, VxFS
IBM AIX

HFS, JFS, JFS2

Not supported: GPFS
Linux for x86 ext2, ext3, reiserfs
Linux for IBM S/390 and zSeries ext2, ext3, reiserfs
Solaris SPARC UFS, VxFS (Veritas), ZFS

Unfortunately not Mac OS X (HFS+ file system).


Macintosh File Systems - HFS...

 

         
Macintosh Mac OS Extended (Journaled)      
  Mac OS Extended      
  Unix File System      
Windows XP NTFS      
Optical Disks ISO 9660      

Mac OS X v10.3.9, Disk Utility v10.4.4

http://www.apple.com/downloads/macosx/system_disk_utilities/

Feature

HFS

HFS Plus

Benefit/Comment

User visible name

Mac OS Standard

Mac OS Extended

Number of allocation blocks

16 bits worth

32 bits worth

Radical decrease in disk space used on large volumes, and a larger number of files per volume.

Long file names

31 characters

255 characters

Obvious user benefit; also improves cross-platform compatibility

File name encoding

MacRoman

Unicode

Allows for international-friendly file names, including mixed script names

File/folder attributes

Support for fixed size attributes (FileInfo and ExtendedFileInfo)

Allows for future meta-data extensions

Future systems may use metadata for a richer Finder experience

OS startup support

System Folder ID

Also supports a dedicated startup file

May help non-Mac OS systems to boot from HFS Plus volumes

catalog node size

512 bytes

4 KB

Maintains efficiency in the face of the other changes. (This larger catalog node size is due to the much longer file names [512 bytes as opposed to 32 bytes], and larger catalog records (because of more/larger fields)).

Maximum file size

231 bytes

263 bytes

Obvious user benefit, especially for multimedia content creators.

 

 


Windows File Systems - FAT, NTFS...

First introduced FAT32 in Windows 95 OSR2 (OEM Service Release 2) and then later in Windows 98. NT-based Windows did not have *any* support for FAT32 up to Windows NT4; Windows 2000 were the first NT-based Windows OS, that received the ability to work with it.

Windows - NTFS or FAT

NTFS or FAT www.adrc.com/ckr/ntfs_fat.html (local)

Another comparison - www.ntfs.com/ntfs_vs_fat.htm (local)

Criteria

NTFS5

NTFS

FAT32

FAT16

FAT12

Operting System

Windows 2000
Windows XP
Windows 2003 Server

Windows NT
Windows 2000
Windows XP
Windows 2003 Server

DOS v7 and higher
Windows 98
Windows ME
Windows 2000
Windows XP

DOS
All versions of
Microsoft Windows

DOS
All versions of
Microsoft Windows

 

Limitations

Max Volume Size

2TB

2TB

32GB for all OS.
2TB for some OS

2GB for all OS.
4GB for some OS

16MB

Max Files on Volume

Nearly Unlimited

Nearly Unlimited

4194304

65536

Max File Size

Limit Only by
Volume Size

Limit Only by
Volume Size

4GB minus 2 Bytes

2GB (Limit Only
by Volume Size)

16MB (Limit Only
by Volume Size)

Max Clusters Number

Nearly Unlimited

Nearly Unlimited

4177918

65520

4080

Max File Name Length

Up to 255

Up to 255

Up to 255

Standard - 8.3
Extended - up to 255

Up to 254

 

File System Features

Unicode File Names

Unicode Character Set

Unicode Character Set

System Character Set

System Character Set

System Character Set

System Records Mirror

MFT Mirror File

MFT Mirror File

Second Copy of  FAT

Second Copy of  FAT

Second Copy of  FAT

Boot Sector Location

First and Last Sectors

First and Last Sectors

First Sector and
Copy in Sector #6

First Sector

First Sector

File Attributes

Standard and Custom

Standard and Custom

Standard Set

Standard Set

Standard Set

Alternate Streams

Yes

Yes

No

No

No

Compression

Yes

Yes

No

No

No

Encryption

Yes

No

No

No

No

Object Permissions

Yes

Yes

No

No

No

Disk Quotas

Yes

No

No

No

No

Sparse Files

Yes

No

No

No

No

Reparse Points

Yes

No

No

No

No

Volume Mount Points

Yes

No

No

No

No

 

Overall Performance

Built-In Security

Yes

Yes

No

No

No

Recoverability

Yes

Yes

No

No

No

Performance

Low on small volumes
High on Large

Low on small volumes 
High on Large

High on small volumes
Low on large

Highest on small volumes
Low on large

High

Disk Space Economy

Max

Max

Average

Minimal on large volumes

Max

Fault Tolerance

Max

Max

Minimal

Average

Average

 

 

 

 

 

  

 

 

 

 

 

  

 

 


ZFS

http://blogs.zdnet.com/storage/?p=663&tag=content;col2

October 25th, 2009

Mac ZFS is dead: RIP.

Posted by Robin Harris @ 11:34 pm

Categories: Infrastructure, Software

Tags: NetApp Inc., Apple Macintosh, Sun Microsystems Inc., File System, Apple Inc....

PC file system progress stalled this week with the news on MacOSforge that Apple’s ZFS project is dead.

ZFS Project Shutdown 2009-10-23
The ZFS project has been discontinued. The mailing list and repository will also be removed shortly.

ZFS, developed by Sun engineers, is the first 21st century file system. NTFS and HFS+ are firmly rooted in the 1980s. ZFS has a lot of cool features:

  • End-to-end data integrity. Current file systems are prone to many problems - ranging from phantom writes to inconsistent error-handling - that mess up your data. The ZFS architecture eliminated them with parent block checksums.
  • Pooled storage. Add a drive and it adds extra capacity, not another volume. Less management.
  • No need for journaling. Which is one problem Solid State Drives don’t handle well. Get rid of it and SSDs work better.
  • Built-in RAID that is as fast as hardware RAID. Get data protection for a lower cost.
  • Low-cost snapshot copy. As a copy-on-write system, ZFS can create new snapshots - once an hour, minute or second - with low CPU and storage overhead. Cruise back in time to just before the virus hit, recover, and life is good.

Apple announced in June ‘08 that Snow Leopard server would support ZFS. But things came apart early this year.

What happened?
Jeff Bonwick, ZFS architect, posted Saturday on an earlier quoted comment:

> Apple can currently just take the ZFS CDDL code and incorporate it
> (like they did with DTrace), but it may be that they wanted a “private
> license” from Sun (with appropriate technical support and
> indemnification), and the two entities couldn’t come to mutually
> agreeable terms.

I cannot disclose details, but that is the essence of it.

Jeff

Indemnification?
Sun is being sued by NetApp, a $3B enterprise storage company, claiming that ZFS infringes on NetApp patents. If NetApp won, Apple would find itself in a tough position unless Sun shouldered the financial damage. That’s indemnification.

Sun has made a (IMHO) strong case that NetApp’s patents should be invalidated by prior art. But with all their other problems and the Oracle purchase it was a headache they and Oracle didn’t need.

Where does Apple go from here?
Apple has hired some smart file system engineers and wants to hire more to work on “state-of-the-art file system technologies for Mac OS X.”

But writing new file systems isn’t easy. It takes 5-7 years for a new file system to achieve the maturity needed to support large-scale deployment.

So if Apple is starting from scratch we have a long wait for real innovation to appear. Like Mac OS XII.

What about Microsoft?
Redmond’s file system gurus are well aware of NTFS issues. And under the covers they are making stepwise enhancements to the architecture and implementation.

But as the NTFS and HFS+ architectures age and the pace of storage innovation increases the gap between what is and what could be grows. It’s like putting a 1001 hp Bugatti engine in a Model T: the power is there but you can’t use it.

The Storage Bits take
This kind of cock-up makes me hate software patents - but that’s another post. As long as law allows companies will try to enforce them.

NetApp missed a golden opportunity to raise their visibility in the consumer market by cutting a deal with Apple directly. “NetApp is powering Apple’s advanced storage technologies” would make the company a lot more visible outside the enterprise market.

NetApp is a good company, but they’ve lost their way lately. Note to new CEO Tom Georgens: with EMC moving aggressively into the consumer space you don’t have forever to reposition NetApp for a consumer-driven world.

Steve Jobs doesn’t get storage. Consumers are generating masses of video and photos at an accelerating pace - and they’ll need reliable, available and dirt-easy storage. Lots of it.

Until the Next New Thing in file systems rolls out of Cupertino, Redmond or, maybe, Redwood City, consumers will stuck with too many BSODs, missing and corrupted files and app crashes. Let’s hope we don’t have to wait too many more years.

Comments welcome, of course. Update:There’s now a Google Code page for MacZFS.

 

 

Date Notes
20061021 First version
20090311 CSS Frames, cleaned up
20090714 Added section on ClearCase
20111020 Minor cleanups

This section uses Nov 08 CSS Frames.
JandP.biz : J&P Group | J&P BMCG | J&P InfoSol | J&P INV |J&P Multimedia Labs | J&P NoBoS Travel
  Contact us | Copyright © 2000-2020 J&P Group. All rights reserved.
Technology:
1&1