by Stefano Maffulli | Sep 4, 2018 | Cloud Storage, Tutorials
It’s easy to make mistakes when developing multi-cloud applications, even when only dealing with object storage API. Amazon S3 and Azure Blob Storage are similar models but with differing semantics and APIs, just like Google Cloud Storage API. Amazon S3 is a RESTful API providing command syntax for create (PUT), access (GET), and delete (DELETE) operations on both buckets and objects, plus access to bucket metadata (HEAD).
Applications that need to support both API have to be developed very carefully to manage all corner cases and different implementations of the clouds. Luckily, Zenko’s team is dedicated to finding those corner cases and solve them once for everybody. Zenko CloudServer translates standard Amazon S3 calls to Azure Blob Storage, abstracting complexity. The design philosophy of CloudServer’s translations are:
- S3 API calls follow the Amazon S3 API specification for mandatory and optional headers, and for response and error codes.
- The Azure Blob Storage container is created when the application calls S3 PUT bucket, and the container is assigned the name given in the PUT bucket request.
- Bucket names must follow AWS naming conventions and limitations.
Non-exhaustive API Comparison: AWS versus Azure
Put Bucket / Create Container
- Bucket naming restrictions are similar but not the same.
- CloudServer returns an InvalidBucketName error for a bucket name with “.” even though allowed on AWS S3.
- Canned ACLs can be sent as part of the header in an AWS S3 bucket put call.
- CloudServer uses Azure metadata x-ms-meta-scality_md_x-amz-acl header to store canned ACLs in Azure containers.
Get Bucket / List Blobs
- The AWS S3 “Marker” parameter expects a object key value, but Azure does not have an implementation to retrieve object listings after a certain key name alphabetically (can only retrieve blobs after an opaque continuation token).
- AWS S3 sends back the object owner in each listing entry XML but Azure does not include object owner information in listings.
Delete Bucket / Delete Container
- While AWS S3 returns an error if a bucket is non-empty, Azure deletes containers regardless of contents. Zenko CloudServer makes a call to lists blobs in the container first and returns the AWS S3 BucketNotEmpty error if not empty.
Put Object / Put Blob
- CloudServer only allows canned ACLs, except aws-exec-read and log-delivery-write. ACLs are stored as blob metadata. From the Azure side, there are no object ACLs so behavior is based on container settings.
- Only the STANDARD setting is allowed as “storage class”
- Setting object-level encryption is not allowed through headers. The user must set encryption through Azure on an account basis.
Delete Object / Delete Blob
- AWS S3 has delete versions and offers an MFA requirement for delete. MFA header is not supported in CloudServer.
Get Service / ListContainers
- AWS S3 returns a creation date in its listing, while Azure only stores the last-modified date.
Initiate Multi-part Upload (MPU) / no correspondent on Azure
- A MPU is treated as a regular Put Blob call in Azure. CloudServer cannot allow users to initiate more than one MPU at a time because there is no way of renaming or copying a committed block blob to the correct name efficiently, and any uncommitted blocks on a blob are deleted when the block blob is committed (preventing an upload to the same key name). To allow for initiate MPU, Zenko CloudServer creates a “hidden” blob with a unique prefix that is used for saving the metadata/ACL/storage class/encryption of the future objectListing of ongoing MPUs.
Put Part / Put Block
- Azure has a size limit of 100 MB per block blob. AWS S3 has a max part size of 5 GB.
- Azure also has a 50,000-block maximum. At 100 MB max per block, this comes out to around 5 TB, which is the maximum size for an AWS S3 MPU. Putting the same part number to an MPU multiple times may also risk running out of blocks before 5 TB size limit is reached.
The easiest way to write multi-cloud applications is to use the open source projects Zenko and Zenko CloudServer.
Photo by Simon Buchou on Unsplash
by Stefano Maffulli | Aug 29, 2018 | News
Today Scality announces the first stable release of MetalK8s, the open source K8s distribution focused on bare-metal deployments, long-term maintenance and ease of operation. To bolster these efforts, it joined the Linux Foundation and Cloud Native Computing Foundation (CNCF) as a Silver member.
As the number of organizations using multi-cloud environments continues to increase, Kubernetes is becoming a standard way to manage applications. Scality has invested time and resources over the past 18 months to find the best way to deploy and manage its next-generation product line, and Kubernetes emerged as a clear winner.
It’s exciting to deepen our open source strategy by joining the Linux Foundation and be active with CNCF. With our flagship open source project Zenko we’re incessantly building a strong community and with MetalK8s reaching v.1 we’re hoping to get more people excited about all of our cloud projects.
The team was looking for the best solutions to manage Zenko on-premise for large customers. Early versions of Zenko employed Docker Swarm but the limits in that approach became quickly apparent. Looking at other K8s implementations, no option emerged as a clear winner, so the team decided to build a new solution based on other open source projects.
MetalK8s has a strong foundation with the open source installer Kubespray and other tools, like Prometheus and Grafana. However, because the Scality team has parsed the often baffling options, MetalK8s’ deployments only require few key decisions from operators. The result is a simplified deployment path for a new Kubernetes cluster on bare-metal, with easier long-term maintenance.
People will have strong feelings about the choices made to simplify Kubernetes deployments. That’s why we like to call MetalK8s the opinionated Kubernetes distribution.
MetalK8s version 1.0 comes with default dashboards that help operators keep control of the cluster based on the following services:
- Kubernetes dashboard A general purpose, web-based UI for Kubernetes clusters
- Grafana Monitoring dashboards for cluster services
- Cerebro An administration and monitoring console for Elasticsearch clusters
- Kibana A search console for logs indexed in Elasticsearch
Deployments using release 1.0 can be upgraded to later MetalK8s 1.x versions. The upcoming Zenko version 1 is deployed on a Kubernetes cluster by default: the enterprise edition will initially support only MetalK8s clusters. The open source code has no such limitations and can run on existing or managed clusters like GKE, simply using the same Helm packages.
The MetalK8s GitHub project welcomes contributions, code and documentation as well as issues and questions on the forum.
by Stefano Maffulli | Aug 24, 2018 | Stories
Zenko Connect for Azure enables developers to immediately consume Azure Blob Storage with Amazon S3 applications without any application modifications. Based on the open source Zenko CloudServer code, it’s a free and easy tool to jump from S3 to Azure Blob Storage quickly.
Zenko Connect for Azure provides an Amazon Web Services (AWS) S3 API-compatible front end translator to Microsoft’s cloud storage service, Azure Blob Storage. The core capability of Zenko Connect is translation of S3 API calls into Azure Blob Storage API calls, for application-driven operations on S3 Buckets and Objects. This enables S3-enabled applications to access Azure Blob Storage services natively, without changing their storage API calls.
Zenko Connect for Azure is offered as a free application in the Microsoft Azure Marketplace (the only charges are for Azure infrastructure costs).
Zenko Connect is a stateless service. It maintains and stores all data and metadata in the associated Azure Blob Storage account. The advantages of this stateless model are the capability for scale-out, load balancing, and simplified failover capabilities.
Zenko Connect maps Amazon S3 buckets to Azure Blob Storage accounts and containers. As an application creates S3 objects in an S3 bucket, Zenko Connect stores them as blobs in the associated Azure Blob Storage account or container.
In this release of Zenko Connect for Azure (v. 1.0), API support focuses on:
- Core create, read, update, and delete (CRUD) S3 operations on buckets and objects
- Efficient upload of large objects through the S3 multi-part upload (MPU) APIs
Check the full Zenko Connect for Azure startup guide and full documentation to learn more.
Photo by rawpixel on Unsplash
by Stefano Maffulli | Aug 8, 2018 | Data management, Tutorials
Storing data in multiple clouds without a global metadata search engine is like storing wine bottles without labels in random shelves: the wine may be safe but you’ll never know which bottle will be appropriate for dinner. Using one object-based storage system can easily become complex but when you start uploading files to multiple clouds things can become an inextricable mess where nobody knows what is stored where. The good thing of object store is that objects are usually stored with metadata to describe them. For example, a video production company can include details to indicate that a video file is “production ready” or contain details about the department that produced the file, when raw footage was taken or the rockstar featured in a video. The tags we used to identify pictures of melons with Machine Box example are metadata, too.
Zenko offers a way to search metadata on objects stored across any cloud: whether your files are in Azure, Google Cloud, Amazon, Wasabi, Digital Ocean or Scality RING, you’ll be able to find all the videos classified for production or all the images of water melons.
The global metadata search capability is one of the core design principles of Zenko: one endpoint to control all your data, regardless of where it’s stored. The first implementation was using Apache Spark but the team realized it wasn’t performing as expected and switched to MongDB. Metadata searches can be performed from the command line or from the Orbit graphical user interface. Both searches use a common SQL-like syntax to drive a MongoDB search.
The Metadata Search feature expands on the standard GET Bucket S3 API. It allows users to conduct metadata searches by adding the custom Zenko querystring parameter, search. The search parameter is structured as a pseudo-SQL WHERE clause and supports basic SQL operators. For example, “A=1 AND B=2 OR C=3”. More complex queries can also be made using nesting operators, “(” and “)”.
The search process is as follows:
1. Zenko receives a GET request containing a search parameter:
GET /bucketname?search=key%3Dsearch-item HTTP/1.1
Host: 127.0.0.1:8000
Date: Wed, 18 Oct 2018 17:50:00 GMT
Authorization: <authorization string>
2. CloudServer parses and validates the search string: If the search string is invalid, CloudServer returns an InvalidArgument error. If the search string is valid, CloudServer parses it and generates an abstract syntax tree (AST).
3. CloudServer passes the AST to the MongoDB backend as the query filter for retrieving objects in a bucket that satisfies the requested search conditions.
4. CloudServer parses the filtered results and returns them as the response. Search results are structured the same as GET Bucket results:
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>bucketname</Name>
<Prefix/>
<Marker/>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>objectKey</Key>
<LastModified>2018-04-19T18:31:49.426Z</LastModified>
<ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag>
<Size>0</Size>
<Owner>
<ID>79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be</ID>
<DisplayName>Bart</DisplayName>
</Owner>
<StorageClass>STANDARD</StorageClass>
</Contents>
<Contents>
...
</Contents>
</ListBucketResult>
You can perform metadata searches by entering a search in the Orbit Search tool or using the search_bucket tool. The S3 Search tool is an API extension to the AWS S3 search syntax. S3 Search is MongoDB-native, and addresses the S3 search through queries encapsulated in a SQL WHERE predicate. It uses Perl-Compatible Regular Expression (PCRE) search syntax. In the following examples, Zenko is accessible on endpoint http://127.0.0.1:8000 and contains the bucket zenkobucket.
Search for objects with metadata “blue”:
$ node bin/search_bucket -a accessKey1 -k verySecretKey1 -b zenkobucket -q "x-amz-meta-color=blue" -h 127.0.0.1 -p 8000
Search for objects tagged with “type=color”:
$ node bin/search_bucket -a accessKey1 -k verySecretKey1 -b zenkobucket -q "tags.type=color" -h 127.0.0.1 -p 8000
Search for objects modified on March 23, 2018:
$ node bin/search_bucket -a accessKey1 -k verySecretKey1 -b testbucket -q "`last-modified` LIKE "2018-03-23.*"" -h 127.0.0.1 -p 8000
Zenko’s global metadata search capabilities play a fundamental role in guaranteeing your freedom to choose the best cloud storage solution while keeping control of your data.
Photo by Nick Karvounis on Unsplash
by Stefano Maffulli | Jul 24, 2018 | News
As part of Zenko’s commitment to enable multi-cloud data control, we evaluated Apache Spark as the tool to perform metadata search across multiple clouds. Spark promises ad-hoc queries on arbitrary data without prior indexation, it sounded perfect for the need of a petabyte-scale multi-cloud storage. However, we came to realize that Spark expects semi-structured data, which prevents ad-hoc queries and is not exactly arbitrary data. In our tests we realized that parquet files need to be updated and Spark couldn’t handle raw metadata wouldn’t work, forcing us to post-process it to parquet files. We realized that we were using Spark the wrong way and decided to redesign Zenko metadata system with MongoDB.
Evaluate Spark Performance For Zenko Metadata Search
Zenko architecture team setup a series of tests to evaluate Spark’s performance to enable Zenko’s metadata search capabilities. The lab was setup on our internal OpenStack-powered Scality Cloud, with five 8-core instances called swarm1, swarm2, swarm3, swarm4 and swarm5 running CentOS 7.2. Each of the instance has 50GB of memory except swarm1 which has 64GB. The machines run Scality S3 Connector and Docker Swarm is used to launch Spark version 2.1.0 master and workers. For the tests we used Hadoop version 2.7.3.
Process:
To test load time:
- Call json.read with proper s3a path, provide schema and call cache.
- Do one sql query and call show so have action that will trigger actual read.
- Note that need some action so data will actually be read. Did a little testing and calling count() as the action seemed to be similar time as calling a query and show so did test with query and show.
val t0 = System.currentTimeMillis()
val translatedDF = spark.read.schema(sstSchema).json("s3a://BUCKETNAME/stuff").cache()
translatedDF.createOrReplaceTempView("translatedMD")
val getcertain = spark.sql("SELECT * FROM translatedMD WHERE userMd['x-amz-meta-someotherkey']='someOtherMetadatapartof100millionput977927'")
getcertain.show(10, false)
val t1 = System.currentTimeMillis()
println("Elapsed time: " + (t1 - t0) + "ms")
translatedDF.unpersist()
To test query time:
- Do sql query and call show.
- Command:
var x = 0
for( x <- 1 to 100 ){
val t2 = System.currentTimeMillis()
val getcertain2 = spark.sql("SELECT * FROM translatedMD WHERE userMd['x-amz-meta-someotherkey']='someOtherMetadatapartof100millionput977926'")
getcertain2.show(10, true)
val t3 = System.currentTimeMillis()
println("Elapsed time: " + (t3 - t2) + "ms")
}
Caveats:
- The sst and log files are pre-translated to line delimited json so time to do that work is not accounted for.
- We are not filtering for duplicates based on incrementer so time to do that work (which will likely involve a shuffle) is not accounted for.
- If lose workers when loading data, subsequent query will take longer. For instance, with 100 million objects, lost 2 workers during read. Subsequent query then took around 60 seconds.
- Numbers reflect accessing data through one s3 instance.
Results
Metadata for 1 million objects (all numbers are average of 10 runs)
|
1 worker |
5 workers |
5 workers parquet |
10 workers |
Read time in sec |
14.24 sec |
15.59 sec |
12.66 sec |
17.63 sec |
Query time in ms |
260 ms |
197 ms |
196 ms |
258 ms |
Memory used on worker for read |
4.82 GB |
1.66 GB |
1.23 GB |
1.15 GB |
Memory used on master for read |
< 1 GB |
1.14 GB |
1.11 GB |
1.21 GB |
Metadata for 10 million objects (all numbers are average of 10 runs)
Note that the read times may be artificially low here since the initial read upon starting the spark shell takes longer. The initial read upon starting the spark shell was 44 seconds (1 worker), 41 seconds (5 workers) and 52 seconds (10 workers). For the tests of 1 million and 100 million, the numbers all reflect a read upon starting the Spark shell.
|
1 worker |
5 workers |
5 workers Parquet |
10 workers |
Read time in sec |
34.1 sec |
30.42 sec |
18.507 sec |
37.76 sec |
Query time in ms |
575 ms |
376 ms |
342 ms |
360 ms |
Memory used on worker for read |
20.09 GB |
6.43 GB |
5.11 GB |
5.97 GB |
Memory used on master for read |
< 1 GB |
1.21 GB |
1.47 GB |
1.3 GB |
Metadata for 100 million objects (all numbers are average of 10 runs)
|
1 worker |
5 workers |
5 workers parquet |
10 workers |
Read time in sec |
348.6 sec
(5.81 min) |
184.02 sec
(3.07 min) |
73.842 sec (1.23 min) |
281.4 sec
(4.69 min) |
Query time in ms |
72,110 ms
(72.110 sec) |
1,745 ms
(1.745 sec) |
1,784 ms
(1.784 sec) |
2,165 ms
(2.165 sec) |
Memory used on worker for read |
36.76 GB |
24.37 GB |
23.86 GB |
14.03 GB |
Memory used on master for read |
< 1 GB |
1.18 GB |
1.13 GB |
1.28 GB |
Conclusions
These numbers convinced us that we were on the wrong path and the team needed to go back to the drafting board. The tests showed that Spark needed a lot of resources to function properly in our scenario, which forced a very high latency due to the need to reload all parquet files every time there was a change. We realized that we were trying to force Spark into an unnatural role, so we switched Zenko to a new design based on MongoDB.