Filesystems

In the last months, I noticed that it was a bad decision to use XFS as my filesystem; and before I realized that ext3 was a bad choice as well.

File systems have problems. I can’t use ext3 on my laptop, because it would check the whole disk (120GB) every 30 boots. And this check takes a very long time (15 or 30 minutes, I do not know it exactly). XFS is better, but also problematic because deleting a big directory with a deep hierarchy takes a very long time like 1.5 minutes , whereas ext3 needs a few seconds. Also I still wonder why XFS needs no checks.

Therefore I might go with ext4. The time needed to check the partition seemed to be much shorter and the speed is about the same as with ext3. The installation will probably be: Install Lenny on a ext3 partition, install kernel with ext4 support (which is kernel 2.6.28), add ext4 to /etc/initramfs-tools/modules, tar the partition, format it as ext4, and untar the contents again; because converting does not bring the advantags of ext4 to already existing files.

BTW, I did some benchmarking with bonnie++, and XFS had about 20,000 operations per second, whereas ext3 had 60,000. ext4dev from Kernel 2.6.27 was a bit slower than ext3

I also tested BTRFS, but BTRFS is 1) not stable enough 2) as slow as XFS. It also shows horrible write performance of 19MB/s using dd, whereas XFS reached 42MB/s. BTW, benchmarking btrfs with bonnie++ only works using “-s 0”, i.e. without I/O performance test, else you receive a kernel bug. I subscribed to the btrfs mailing list, and sent my bug report email 4 times, but it never reached the list it seems. This bug also happens in other cases.

26 thoughts on “Filesystems

  1. Another three big advantages of xfs over ext3 are the ability to defrag your hdd (without unmounting btw), the ability to grow the filesystem while it is mounted (comes in handy combined with LVM) and the better handling when space is running low (ext3 is terrible in that case).

  2. On my sid laptop, the fsck checks are skipped if I’m on battery power. So it’s not really that big a deal.

    If I really want to skip the check, I just reboot and pull the power as it checks whether I’m on AC or not.

    1. Well, disabling or decreasing the check interval might solve the problem. But there are reasons for file system checks, which also confuses me because XFS filesystems are normally not checked.

  3. If it bothers you so much and you aren’t overly worried about your data which you can’t be if your thinking of using ext4. Try

    tune2fs -c 0 -i 0 /dev/sda1

  4. The different of XFS/btrfs/ext4 over ext3 is that they actually disable write barriers and cache flushing on fsync by default. This is the only way to ensure a filesystem is not actually corrupted on an unclean shutdown with consumer disks that have the write cache enabled. Note that at least for XFS it can actually be much faster to just disable the write cache and barriers instead (hdparm -W 0 /dev/sda, needs to be done on every boot as it doesn’t stick).

    ext3 wouldn’t need the fs check on boot if it actually deal with barriers properly, but even ext4 which doesn’t need it either has it enabled for some odd reason.

  5. For the ext4 transition with on Debian, you should also edit your /usr/share/initramfs-tools/scripts/local file, as it make the initramfs detect the root partition’s filesystem using fstype, which still (as of Debian unstable) doesn’t recognize ext4, nor ext4dev. Udev’s vol_id recognizes more filesystem types.

    Edit it as this: http://pastebin.com/m7e90495e

    Then run:

    initramfs -u -k ${KERNEL_VERSION}

    for your ext4 capable kernel.

  6. Sorry to hear your messages aren’t getting through to the btrfs list. They aren’t in the archive as far as I can tell, it is likely the vger.kernel.org spam filters are catching them.

    The oops you mentioned from the btrfs list was the disk filling up. I’d be interested in seeing the oops you did see on your box.

    Ext3 and Ext4 default to doing an fsck every 30 days because the e2fsprogs maintainer feels that is a good general practice. The theory is that hardware is unreliable and even though the filesystem is crash safe, an additional check can be a good idea.

    In practice, XFS is no more or less crash safe than ext3 and ext4, aside from the XFS policy of enabling cache flushing by default (which is important on sata drives).

    Using tune2fs to disable the periodic fsck on ext3/4 is safe. I do recommend enabling barriers on ext3 with mount -o barrier=1

  7. Hey Julian,

    I must admit that I’m happy with ext3 on all of my laptops, one has a 100G drive, and checks are done in <1min, and none of them do the checks when on battery, because powermgmt-base is installed.

    Regards
    Evgeni

  8. A good balance between filesystem safety and boot time is to do the filesystem check after n days instead of n (re)boots. I realy do not get why Ubuntu doesn’t change this by default, because on all non-server machines a filesystem ckeck every 30 boots is just insane.

    tune2fs -c 0 -i 30 /dev/sdaX

    would change the filesystem check intervall to 30 days. If you have more than one partition, you can vary the interval slightly for every partition, so that not all partitions are checked the same time. This way boot time delays should not be a problem at all and the checks are still done regulary (adjust the intervalls to your needs).

  9. I was wondering if you conducted any tests with reiserfs? I’ve always heard that reiserfs is stable and efficient, but I have not used it in a few years (since my last Gentoo build). What’s your take on reiserfs?

  10. I researched a lot about filesystems when I got a new HD, and after experimenting a lot, I ended up with ext3 in my root, which is small and has to work very reliably, and xfs to hold all my data (about 200gb), and it is doing great so far.
    I wouldn’t recommend xfs for root tough, because it can be very slow when creating/updating a lot of metadata, which happens a lot when installing / updating the system, and it made my system update completely hang my system for a few minutes.

    There is a way to make XFS much faster tough, by adding some parameters when the filesystem is created, and when it is mounted. Especially the slowness when deleting directories can be decreased a lot (to the point you don’t even care).
    For instance, this is my mounting parameters (you can read about it in the mount man page): relatime,logbufs=4,logbsize=256k
    You can have more information here about the parameters when creating the filesystem: http://everything2.com/node/1479435
    And you can read the mkfs.xfs man page, which has some great information.

    I am too looking forward to brtfs. Linux do need a ZFS equivalent.
    Btw, ReiserFS sucks! I lost too much data to it already, and the performance gain is not worth at all.

  11. ok, first things first.
    When you say that ext3 needs to be checked every 30 mounts, that is just a setting which can be changed with tune2fs (like someone said earlier). Bear in mind that ext3 is a journaled file system while xfs is not, so if you have any crashes while there are writes in progress, data might be lost.
    Second of all, dd bypasses the filesystem alltogether by accessing the raw sectors on the disk instead of the files/inodes. When you see different results with different file systems you might want to check that there hasn’t been any changes in the meantime (like DMA). All in all, there are pros and cons for every filesystem. While ext3 and especially reiser are very fast to delete large amounts of files, when you delete a few big files, say of 10 GB or more each, xfs really shines.

  12. Perhaps you should look at JFS, the fsck is practically instaneous. I haven’t done any benchmarks, but I find it to be an adequate performer. That is all.

  13. Maybe everyone here is some sort of power user, but for the vast majority of Linux users (laptop, desktop, and server), ext3 seems to work fine.

    Just turn off the auto fsck’s if they bother you that much and add a reminder to your calendar to manually check your disk every month. That way you can do it whenever is convenient.

    I used to use reiserfs years ago when I used slackware (circa 2003?), and it worked fine. Never had any of the “massive data corruption” problems I always hear people complaining about.

    That being said, resierfs isn’t being actively developed anymore, so I don’t think any sane person would choose it over ext3 without a very specific reason (eg some unique feature).

  14. I’ve been having similar troubles myself (see this post of mine), and decided to give ext4dev a go on 2.6.27.

    Fast is fast, and seem to work quite nicely, but I already had it to misbehave, during a tinderboxing run my /var/tmp directory (where Gentoo builds the software) went read-only without notice.

    I’m also looking forward for btrfs.. but just not yet, not yet.

  15. So why don’t you swith off fs checking if you don’t need it? I use ext3 at dozens of machines this way, and sure maybe this is not recommended, but I haven’t experienced any problems yet. XFS in Ubuntu server/generic kernels for example often dumps kernel crashes, I had file system corruptions etc, so I dropped it and I use ext3 now everywhere without any problem.

  16. @ Yoram

    Bear in mind that ext3 is a journaled file system while xfs is not, so if you have any crashes while there are writes in progress, data might be lost.

    Who told you that?

  17. The fsck every N boots used to bother me. Then I got an Intel SSD, and they take about eight seconds.

  18. Ext3 is a safe option for /boot and / because it’s the most commonly used filesystem. The technical difficulties in doing a check of a non-root filesystem are signficantly lower than for a root filesystem (for starters you can have swap enabled so more memory is available for fsck).

    One difference between ext3 and XFS is the maximum write-cache time for data that has not had fsync() called on it, for ext3 it’s 5 seconds and for XFS it’s 30 seconds. I’ve seen a few programs lose data on XFS systems after a power outage because the application programmer didn’t use fsync() when they should – on ext3 they probably would have got lucky.

    As has been noted you can change the frequency of fsck on ext3. I believe that ext3 and XFS are both mature and stable filesystems developed by competent people, in all cases other than a power failure I don’t expect any difference in reliability (I am not qualified to comment on write-barrier issues). If you believe that XFS is safe for a fsck every 90 days without crashes then I expect that you could say the same for ext3!

    Also one thing to note about performance is that XFS is designed for larger filesystems and heavier loads than Ext3. It may be that Ext3 is better for laptops due to engineering trade-offs for supporting big hardware in XFS (NB I am not claiming this as fact but stating it as a possibility). So I think that statements about being “better” should be “better for a laptop”.

    Finally there are some significant differences between Ext3 and XFS in terms of XATTRs. Make sure you use a 512 byte Inode size for XFS if you want to run SE Linux.

  19. We discovered something slightly surprising at work recently. ext3 has three modes for the data (rather than metadata), ordered (default), journalled and writeback.

    Journalled keeps the data in the journal, which is the safest and traditionally the slowest. Writeback does more aggressive caching of the data and is traditionally the fastest, but less safe.

    What we found is that with lots of small writes actually journalled is the fastest. This is because it minimises seeks. When you are writing the file it gets streamed into the sequential log file. Then it can be written from the log at leisure when the seeks don’t matter.

    You could try ext3 in journalled data mode to see if that improves it for you.

  20. I guess you somehow inspired me. Following your post, I decided to update my intrepid installation with a self-compiled Kernel.

    I reformatted my /home to ext4. So far, it’s doing good. Even though I don’t have any serious data about that, it does seem faster.

    Emms

  21. Well…; Due to a kernel bug related to SATA drives the fsck autocheck thing screwed my FS (first the /, then the /home) 2 times, with 2 different kernel versions. Using EXT3 as unique FS. I hope that EXT4 fix that, so i can install Jaunty with that instead of ext3…;

Comments are closed.