class: center, middle # Apache jclouds at Maginatics *experiences with billions of blobs across many blobstore providers* Andrew Gaul, jclouds PMC 4 March 2014 http://jclouds.apache.org/ https://maginatics.com/ http://gaul.org/talks/jclouds-at-maginatics/ --- # Agenda * What is blob storage * Blobstore compatibility * Case study: MagFS * Scaling * Lessons learned * Future directions * Conclusion --- # What is blob storage * blobstores offer key-value storage that is * scalable: 10s TB with few nodes and 100s of PB with thousands of nodes * inexpensive: built on commodity hardware * available/durable: tolerates hardware failures * do *not* offer guarantees that block storage and file systems provide: * limited interface: get, put, delete * eventual consistency: blob reads may return stale or no data for some limited time --- # jclouds supports many providers * multiple public and private implementations allow customer trade-offs
--- # Blobstore compatibility * jclouds abstracts differences between APIs, but semantic differences remain: * Atmos: cannot overwrite blob * AWS-S3: cannot mutate or append to a blob, cannot put blob without explicit size * Swift: eventually consistent * portable applications must use the lowest-common denominator functionality * write to blobs exactly once, never mutate or append * can read from blobs at any time, but must retry due to eventual consistency * when deleting, never reuse blob name --- # Case study: MagFS * virtualized, cloud-based storage system * layers network file system semantics on top of blob storage * run any application on a variety platforms, including multiple-client file sharing * more similar to NetApp filer than Dropbox or s3fs * smart client gives LAN performance over WANs * flexible deployment options: public, private, hybrid cloud * refer to [SNIA SDC 2013 slides](http://www.slideshare.net/ntolia/tolia-snia-sdc2013) for technical background --- # Scaling throughput * MagFS supports thousands of clients reading and writing simultaneously * single server could become a bottleneck, especially smaller instance sizes * instead vend signed URLs to clients to allow them direct access to blobstore * cryptographically signed URLs allow read or write access to a specific blob for a specified time * can embed other properties like content length and hash * this technique allows a single MagFS server to mediate many Gbit/s throughput! --- # Scaling number of blobs * MagFS manages 100 TB of blob data across 1 billion blobs * some providers require specific naming or sharding for best performance * Atmos: no more than 100,000 blobs per directory, shard across directories * AWS-S3: name blob with unique prefixes * Swift: no more than 1 million blobs per container, shard across containers * GCS & HPC: remove Expect: 100-continue * other quirks: Cleversafe performs better when disabling container listing * surprisingly challenging workload: removing all blobs from a large container --- # Scaling blob sizes * most MagFS blobs have small sizes, but some use cases require larger ones * jclouds support up to 2 GB blobs across all blobstores * could support 5 GB with Java 7 * AWS-S3, Azure, and Swift support multi-part upload, tested with 40 GB blobs * large blobs increase chances of transient network errors and failures * use a repeatable Payload like ByteSource to allow jclouds to retry * always include MD5 checksum to guarantee data integrity --- # Lessons learned * cross-provider support required substantial effort * long tail of issues with authentication, configuration, error codes, timeouts, etc. * S3- and Swift-compatible clones are like snowflakes, no two are alike * measuring performance is difficult * blob naming and sharding important * public providers will reshard very active containers for better performance * private blobstores require configuration and tuning * mock blobstores (filesystem and transient) helped testing --- # Future directions * more diagnostic tools, especially for private blobstores * Maginatics will contribute benchmark tool and compatibility tester * modernize with Guava additions, e.g., ByteSource, Hashing, MediaType * simplify implementation * de-async? * remove annotations? * new providers * modernized Swift (in-progress) * Google Cloud Storage (GSoC 2014?) * Amazon Glacier * Joyent Manta? --- # Recap * jclouds can provide portability between blobstore providers if your application does not strongly depend on blobstore semantics * applications can scale with the correct architecture and implementation choices * more work to do to make jclouds an inviting platform for all Java developers * jclouds community helped Maginatics over the last three years and we look forward to continuing to contribute --- class: center, middle # Thank you! http://jclouds.apache.org/ https://maginatics.com/ http://gaul.org/talks/jclouds-at-maginatics/