During the last “Powering the Cloud” event in Frankfurt, I had the opportunity to privately meet some exhibitors. One of them has been Spectra Logic. I had the opportunity to talk directly with Matt Starr, Spectra Logic CTO. It was the perfect way to learn as much as possible about their new solution, just announced recently: Black Pearl.
Spectra Logic Black Pearl
Spectra Logic Black Pearl is a new archiving solution, just launched few weeks ago. It’s and appliance, and is a piece of a wider storage solution called Spectra Logic “Deep Storage”, together with their tape libraries that are since the beginning their top product.
BlackPearl sits between tape libraries and applications writing to them, acting basically as a cache system. Thanks to its 4 100 GB SSD disks, the appliance has two main functions. First, it saves on its own disks data coming from applications, so it can create a sequential buffer to be then sent to tape library. Data sequencing helps to create a continous stream towards tapes, so they do not have to suffer from frequent start/stop/seek operations. Also, the appliance saves all the informations and metadata about all the data it has sent to tapes.
Using a 10G ethernet connection (or an optional Infiniband connection),Black Pearl can receive and send data at a maximum speed of 400 MB/s.
Object Tape Storage
However, the real innovation is not the hardware architecture; there are already on the market several solutions with disk caching used for libraries. The innovation comes in the way Black Pearl is accessed.
Access to the appliance happens only via DS3 APIs, developed directly by Spectra Logic. It’s an API library 100% compatible with S3 APIs from Amazon Web Services, with the addition of three fundamental commands: bulk put, bulk get, eject bucket. These new commands give the possibility to use a tape system as object storage, and only by saying it it’s already cool stuff, isn’t it?
That’s how it works: Black Pearl only uses LTFS to write to tapes. This helps to avoid proprietary formats on tapes, and so be able to mix every type of tape inside the libraries (and enable further use cases I will list later). The single unit for archival in the Black Pearl is a bucket, with the same meaning that it has in Amazon: a collection of objects belonging to a single account. You can compare it to a directory in a file system if you want. Each account can have several buckets,but a single bucket can be owned only by one account. In the library, each tape belongs to only one bucket, and a bucket can be spread over several tapes, based on its size.
Writes and reads on the Black Pearl are done via HTTP REST calls. The advantage is clear: all the protocols designed for disk file systems (CIFS, NFS…) do not allow for the long response times of tapes, and so they will incur in timeout errors. By using HTTP calls the Spectra Logic solution can extend the waiting time virtually to an infinite amount of time, thanks to a really smart “trick”: while the Black Pearl is executing a single operation requested by an application, every 30 seconds its web service issues to the application a redirect to self (HTTP/3XX), thus adding at each cycle 30 seconds. The activity is completed when the web service finally replies with a HTTP/200. Thanks to this the bulk commands can also last several hours, that is the time needed to complete for example the uploads of huge amounts of data into the tape libraries.
Designed for Service Providers
There is no doubt this solution has been designed mainly for Service Providers, and during the meeting I had further validations of this.
The choice to support AWS S3 for example offer a quick and easy migration path for all those application originally designed for Amazon Web Services with a small effort. I’m sure that application like Cloudberry could quickly add the new commands and thus become compatible with Black Pearl. For sure, this is only the biggest limit of this solution: you need to buy or created your own application able to use DS3 libraries in order to use Deep Storage. Developers can have access to a SDK, and also a complete simulator so they do not have to buy a hardware system. At the moment there are already 55 registered developers; it’s only a matter of time before we will see the first application supporting Black Pearl.
Second aspect, the bucket and the way data are saved to tape. Since each tape belongs to only one bucket, it’s really easy and quick to export a bucket and give it to a customer, since it only requires to eject from the tape libraries all the tapes belonging to a given bucket. After the misfortunes of Nirvanix and all the problems it caused to its customers, the possibilty to have a quick solution to move out all your data in a short period of time is a real plus, especially in situations where the overall amount of data is huge.
Think about this scanario: a customer has slowly archived 1 PB of data over time, and at a certain point in time he needs to get them all back. The maximum speed of a Black Pearl is 400 MBs. With this speed, the complete download of all those data will last 694 hours, that is almost 29 days. And there is another problem: the customer almost certainly is not going to have 1 PB of storage, and to get one new storage of that size is not easy at all. But if you think about tapes, a single LTO-6 tape holds up to 2.5 TB. By using an express courier service, you can simply store that Petabyte in a box (1 PB is 400 tapes) and ship it almost anywhere in 24 hours. There is probably no internet connection able to move 1 PB in a single day. Moreover, you can also do data import: a customer can ship its tape to the Service Provider as long as they are using LTFS, and Deep Storage will import the data. Spectra Logic is already planning for the near future the possibility to read also proprietary tape formats created with the most common backup softwares.
Anothert feature designed with service provider in mind is encryption. All the traffic happens over HTTPS connections, and there is also encryption at rest. The encryption keys are managed by the Black Pearl appliance, or directly by the library. When a customer gets its tapes back, the key can be exported and delivered to the customer, so he will be able to decrypt and read his own data.
An alternative to AWS Glacier
In my opinion Black Pearl / Deep Storage is an awesome archival solution, that can be an effective way for a Service Provider to compete with Amazon Glacier. First of all it has a rich set of features as I described in this article. Glacier for example only allows data export up to 16 TB via NAS.
But even in regards to price, a Service Provider can offer a competitive solution. Glacier has become famous for its “1 penny per GB per month” price. But even with Black Pearl you can reach those price, or even less. First, electricity costs for a tape library are almost zero, since they absorb only few watts.
But most of all, the pricing of the solution is really interesting. The smallest Deep Storage solution is comprised of one Black Pearl appliance and a tape library with 1,9 PB storage with dual copy; and its price is 14 cents per GB. If you sell this solution at the same price as Glacier, you will reach the breakeven after 14 months; but obviously this price is even lower if you look at bigger systems. Spectra Logic says the price is below 9 cents if you order a Deep Storage with more than 10 PB. With this price, after only one year a service provider is already profitable.
Final notes
I really liked the Black Pearl / Deep Storage solution. And seems also many potential buyers liked it, since Spectra Logic was already able to mention among the early adopters Yahoo and NASCAR. Matt Starr was not allowed to mention any name, but he confirmed several Service Providers are already evaluating this solution.
I’m sure in a short time, there will be on the market some substantial AWS Glacier competitors, and they will probably use Deep Storage as their archival technology.