Brad Fitzpatrick (bradfitz) wrote in lj_backend,
Brad Fitzpatrick
bradfitz
lj_backend

MogileFS transition

As of tonight, all userpics, phoneposts, and captchas are now stored on our MogileFS file storage system.

Our old system, while well-intentioned, was pretty cheesy and lame technically. It was never meant to be used for long ... it was mostly just a crutch until we figured out what we really wanted to do.

Here's a snapshot of our MogileFS installation at present. We have 6.14 TB free. And if that's not enough, we have 10 machines on-hand that could store 1TB each if we run out of room. We'd just need to throw 4 hot-swap SATA disks in them.
lj@grimace:~$ mogcheck.pl
Checking mogilefsd availability...
        10.0.0.81:7001 ... responding.
        10.0.0.82:7001 ... responding.

Device information...
  hostname     device   age    size(G)       used       free    use%  delay
      sto1       dev1   56s    224.319     15.022    209.297   6.70% 0.004s
      sto1       dev2   56s    229.161      9.337    219.823   4.07% 0.004s
      sto1       dev3   56s    229.161      9.273    219.888   4.05% 0.005s
      sto1       dev4   56s    229.161      9.308    219.853   4.06% 0.004s
      sto1       dev5   56s    229.161      9.271    219.890   4.05% 0.013s
      sto1       dev6   56s    229.161      9.409    219.752   4.11% 0.009s
      sto1       dev7   56s    229.161      9.305    219.856   4.06% 0.005s
      sto1       dev8   56s    229.161      9.342    219.819   4.08% 0.004s
      sto1       dev9   56s    229.161      9.298    219.862   4.06% 0.007s
      sto1      dev10   56s    229.161      9.245    219.916   4.03% 0.008s
      sto1      dev11   56s    229.161      9.334    219.826   4.07% 0.004s
      sto1      dev12   56s    229.161      9.281    219.879   4.05% 0.005s
      sto1      dev13   56s    229.161      9.364    219.797   4.09% 0.006s
      sto1      dev14   56s    229.161      9.295    219.865   4.06% 0.008s
      sto2      dev15   10s    224.319      9.342    214.977   4.16% 0.004s
      sto2      dev16   10s    229.161      9.317    219.843   4.07% 0.006s
      sto2      dev17   10s    229.161      9.394    219.767   4.10% 0.005s
      sto2      dev18   10s    229.161      9.387    219.774   4.10% 0.005s
      sto2      dev19   10s    229.161      9.236    219.925   4.03% 0.004s
      sto2      dev20   10s    229.161      9.312    219.849   4.06% 0.006s
      sto2      dev21   10s    229.161      9.211    219.949   4.02% 0.005s
      sto2      dev22   10s    229.161      9.312    219.849   4.06% 0.010s
      sto2      dev23   10s    229.161      9.231    219.930   4.03% 0.004s
      sto2      dev24   10s    229.161      9.370    219.791   4.09% 0.006s
      sto2      dev25   10s    229.161      9.305    219.856   4.06% 0.008s
      sto2      dev26   10s    229.161      9.243    219.917   4.03% 0.013s
      sto2      dev27   10s    229.161      9.264    219.896   4.04% 0.009s
      sto2      dev28   10s    229.161      9.326    219.834   4.07% 0.004s
                total         6406.817    266.336   6140.481   4.16% 0.173s

Those top two lines are checking on the mogilefsd trackers... they're the servers that keep track of where all the files are at. They're actually just a protocol translator in front of the same MySQL database. And if that database goes down? Well, then we'd be screwed. That's why the database is currently on really nice hardware. But the real plan going forward is to use MySQL Cluster, which we'll be using for our global master DB as well. Then there'd be no single point of failure at all.

Oh, and the MogileFS info shown above is for all of livejournal.com, pics.livejournal.com, and picpix.com.... when you make your MogileFS client object, you just specify what domain you're using. For instance, "danga.com::fb" (for fotobilder) or "danga.com::lj" (livejournal). Then you can have identically named files in all namespaces that don't conflict.

If anybody's interested in using MogileFS, we'd love to help you set it up. Join the list and ask away.
Subscribe
  • Post a new comment

    Error

    Comments allowed for members only

    Anonymous comments are disabled in this journal

    default userpic

    Your IP address will be recorded 

  • 12 comments