Sun and the black box of the internet

by Gary Illyes

SANTA CLARA, CA, United States — What would you do if you would have to store 2 PetaBytes of data? The Internet Archive chose Sun’s Modular Datacenter (Sun MD) equipped with Sun’s Open Storage technology.

The Internet Archive has a massive database, containing 2PB of data, which expands every month with an additional 100TB. The service is based on a very simple concept: it takes snapshots of the known webpages on the internet and stores this snapshot in its massive database. The snapshots are working copies of a webpage rather than an image which makes the service extremely valuable. It is like a library of the WorldWide Web.

The problem of the service is its excellence. Hundreds of thousands of new web pages appear every day which The Internet Archive is meant to index and due to the massive amount of data they have to store, each year The Internet Archive practically had to build a new data center. This year they thought a little bit different: rather than building a new data center, they just opted for a Sun Modular Datacenter (Sun MD) equipped with Sun’s Open Storage technology.

The new Sun MD was installed in March 2009. It is equipped with 60 Sun Fire X4500 (Thumper) Open Storage Systems that run the Solaris 10 OS, including the Solaris ZFS file system. Sun’s servers with Solaris ZFS storage pools enable the Internet Archive to double the storage capacity of its old system while using up to 50 percent less power than other servers would use.

Sun’s Open Storage products are a combination of industry-standard hardware and open-source software, including OpenSolaris, and they can be deployed in Solaris, Linux, Windows, and VMware environments. The latest offering is the family of Sun Storage 7000 Unified Storage Systems, the first open storage appliances. These systems scale from 2TB to 576TB, and add flexibility, ease of use, and energy efficiency to your storage strategy.

The Internet Archive is also using Sun’s new Sun MD remote monitoring service. Sun engineers monitor power, heating and cooling, fire, smoke, and water detection, and physical access points, and will dispatch repair technicians if necessary.