A tiny rant about sqlite

The number of times I've popped up filemon to see Chrome, Safari, or some "Google Safe Browsing" thing thrashing my disk in the last 6 months is kind of a lot.

I usually just kill the offending process. For instance, I think Safari uses some variety of sqlite for its web cache, and once a day it just thrashes for a minute. I mean a full minute during which my computer is basically useless. I have a darned fast RAID on this machine, and it still is the rudest thing in the world.

I'm accusing sqlite. When you have sqlite and it's free, everything looks like an ACID database.

Let me be specific: you do not need ACID for your cache. You can do writeback. You don't need tiny blocks that make my disk seek too much. You can make a big append-only file, and you can tolerate some fragmentation. Be sloppy. Use 2x the space you need.

Stop it.
Posted by on Saturday, December 06, 2008

2 comments:

  1. Are your Chrome, Safari, etc history or cache on RAID5 or similar by chance? Anything that has to calculate parity across a bunch of disks will have a particularly hard time with a bunch of tiny, random writes.

    Sqlite also has a tendency to fsync() or equivalent after every transaction, which Linux's ext3 interacts very poorly with. ( Linux doesn't track which dirty blocks belong to a given open file, so it just flushes them all. Which is kind of expensive to do for Firefox updating its history, etc.) If Webkit offers a way to ask sqlite to not fsync (like Firefox's toolkit.storage.synchronous=0), you might try that.

    -- Aaron
    ReplyDelete
  2. Mostly RAID-1 here (so it's probably not insanely fast), but fsync is probably pretty unnecessary.
    ReplyDelete

Subscribe to: Post Comments (Atom)