The media landscape has changed drastically over the last couple of decades, and will continue to change as technology advances. Companies are looking to lower costs across every part of the production process so how can entertainment creation be made faster and more efficient? As the amount of data continues to grow, companies need a better storage solution.
The following is an excerpt from a live chat between Giorgio Regni, CTO, Paul Speciale, CPO, and Wally McDermid, VP Business Development, all from Scality. The group discussed how cloud workflows can save time and money for the media and entertainment industry. View the chat in its entirety and stay tuned with future ZenkoLive chats on Zenko Forum.
The media and entertainment industry is currently being challenged by four big things
Explosive and unrelenting data growth. The amount and the type of content have certainly drastically changed and continues to change. With that comes complexity for media companies in how they generate, curate and distribute that content.
Data gravity impeding global team collaboration. For example, how do you easily get data, your video files, from where they’re created in Europe, to your creation team maybe in the US, and then distribute it to a market in Asia?. There’s a sense of data gravity and how that slows down the distribution of content internally within a media company.
On-demand consumption model reshaping the established landscape.The way we consume content has also drastically changed. It used to be a single television set and now it’s any number of cell phones and tablets and laptops, and it’s YouTube and it’s Daily Motion and it’s Facebook.
Monetization pressure from new emerging business models
It used to be (to oversimplify it) the media company would send us a television program, insert 2-4 commercials every 10 or 15 minutes, and that was their business model. Now, we subscribe to Netflix. There are ads embedded within videos, or overlaid on top of videos.
Media and entertainment customers don’t want to use the cloud just to back-up or archive – they actually want to process the data.
Paul: It used to just be about one-time data movement, and now it’s much more about active data workflows. You’re really inserting the cloud services into the workflow. It’s kind of a pipeline from capture to delivery. The services are integral to that. And you need to do that in a very efficient manner.
When you talk about media, there’s always a concept of a media asset manager. How does that play with something like Zenko?
Paul: The whole idea of media asset managers is to provide the business intelligence to the end user to catalogue things, to find things. You’re going to have thousands and millions of assets spread around your on-premises system. But now imagine the idea that you have it not only on-premises but in two or three clouds.
The idea of using an intelligent metadata manager like Zenko to tag the metadata and to be able to have the application and Zenko interplay to do things like intelligent searches, seems to be a perfect marriage.
Customers are very concerned about data transfer costs. How cost-effective is a solution like Zenko?
Wally: If you are actually moving less data around the world overall, because you have a global namespace, that will certainly save you network costs.Even if it’s internal data movement from your origin server in the US to, for example, an origin server in Europe. You’ll save money just on the transfer cost across your network. But you’ll also make your workers more efficient, which is a slightly softer cost.
Content is being created so quickly, and needs to get pushed out to consumers so quickly, that anything media and entertainment companies can do to become more efficient and faster in their workflows will save them money/increase their revenue.
It used to be that the standard workflow was to start on-premises, and then flow the data to the cloud as needed. But we’re starting to hear kind of the reverse – where the origin of the storage is in the cloud.
Wally: The larger media and entertainment companies that have an existing on-premises infrastructure and data center are probably going to go on-prem to cloud. But some of the smaller companies who may not have that same expansive footprint on-prem will likely start in the cloud, do much of their work, and then just bring the important summary results back down on-prem.
Paul: The idea is to use cloud first. It’s probably the right place for collection if you have a distributed team. If you’re doing collaboration with teams across the planet, it makes sense.
There’s the old cliché that with great power comes great responsibility. I think sometimes with the cloud providers, it’s ‘with great power comes great complexity.’ I think that’s the challenge that Zenko aims to solve.
Is there a way that people can benefit from the cloud and also enable existing data to be sent to one of the big cloud providers for processing?
Paul: One of the things we realized that can be done through Zenko is to discover that data. We can discover it, we can basically assess the file system, and not move the data but import the metadata into the Zenko name space.
It’s a lighter-weight operation. It doesn’t incur movement of the heavier assets. Once it’s captured in Zenko, there is the ability to apply policies to it. Some of those policies could be to replicate the data to the cloud, to move it to the cloud. It becomes part of the cloud workflow.
Wally: Nobody likes a whole-scale lift-and-shift. Lots of companies have hundreds of filers with a bunch of data on them. The solution we’re talking about simply ingests and creates metadata from your existing data, which can remain in place for as long as you want. Over time you can use it and/or migrate it to different platforms, and you can do that on your own schedule.
How does the cloud impact a collaboration workflow?
Paul: So much of the media industry has been focused around how to move data efficiently between teams. But ultimately, isn’t it better if they’re not moved around?
The cloud provides a global view and global access. But a system like Zenko is even better. Because on top of that it can abstract a global namespace. That global namespace will not only consist of one cloud, it may be multiple clouds, and as we just talked about here it can also include the on-premises systems. I think from a sharing and collaboration perspective, having a global namespace – there’s nothing better than that.
Zenko engineering team has been working for almost two years on the open source code that allows developers to keep control of their data while gaining freedom to choose the best storage options. You’re not forced to pick among Amazon S3, Azure Blob Storage, Google Cloud Storage but you can have them all with Zenko!
ICYMI, watch the recording below to learn why CIOs and data managers love Zenko:
Stores all data unmodified so that it can be accessed and acted on in the cloud directly.
Enables data mobility so that data can be easily placed in the most efficient location.
Provides a single endpoint through which data can be stored, retrieved and searched across any location.
Enables unified data management from anywhere through a secure cloud portal.
Can be deployed using Kubernetes orchestration framework.
Questions from the webinar:
Q. Does Zenko have any plans to integrate with decentralized cloud storage layers?
A. Zenko’s plan is to support as many storage backends as possible and that includes decentralized cloud storage. More specifically, we partnered with Storj and are working to integrate Zenko with their next-generation network.
Q. What is the Orbit website?
A. Orbit is Zenko’s configuration and management interface. Offered as a SaaS graphical interface, it can connect to Zenko instances deployed on-premises or in the cloud. Orbit also offers the possibility to get a free Zenko sandbox to quickly test the capabilities of the system
Q. Does Zenko have its own meta repository to track key/value (metadata) on objects?
A. Object’s metadata are stored in MongoDb inside Zenko cluster and natively stored with the object in each cloud destination.
Q. We’re have a large enterprise class object store with over 10+ PB of data in over 1000 buckets. It’s all internal facing (internal applications/developers/archives, etc). Would Zenko sit in front of this object store (become the primary S3 endpoint) for all apps/consumers using our object store?
A. Zenko can be in front or on the side, depending on the needs of users. Depending on the features of the existing object store, applications can keep on using the existing store while Zenko gets notified out-of-band of updates in the content of buckets, for example.
Q. Can zenko be used to migrate data between clouds?
For existing data, you have to tell Zenko that the data exists. Once the out-of-band updates features is complete, migrations of data can be made automatic.
Q. Does Zenko have its own lifecycle (expiration for example) of objects? Example would be our own back-end object store does not have expiration. Would Zenko be able to delete the object upon expiration?
A. Yes, Zenko can augment the features of an existing object store and we’re seeing requests related to Ceph, which doesn’t have native cloud replication or expiration policies.
Q. Can Zenko keep up with a high request rate s3 customer? (such as 100+ new objects per second)
A. Zenko can handle millions of requests per second, it’s well tested with Cloudserver.
Q. Is there support for IBM Cloud Object Storage (back-end) on prem ; or public?
A. Since IBM Cloud Object Storage is compatible with S3 APIs the integration is quite simple, it requires simple code like we’ve done with Wasabi or Digital Ocean Spaces.
Q. If I connect my instance to Orbit and later disconnect it, what happens to my data and metadata?
A. Orbit is only a configuration and management tool. The metadata is stored inside the Zenko instance in a distributed MongoDB database. Once Orbit is detached, Zenko will continue to operate as usual.
It’s always fun to deal with “chaos monkeys”, a barrel full of fun indeed. A lucky bunch of team members at Scality had their dose of monkey fun at the Paris Kubernetes Meetup. While presenting MetalK8s to the group, Julien Girardin, R&D Engineer and Laure Vergeron, Zenko Technology Evangelist, mused over Zenko’s architecture and its resilient design. The meetup was a joint event with the serverless, Docker, and chaos engineering communities, and the keynote speaker was from the Chaos Engineering community.
You get chaos in your systems when there is an urgent problem, with assumed causes and consequences, but the link between them is unclear and so is the path to resolution. The ability of a system to cope with chaos is called resilience. Resilience can be defined as a system’s ability to absorb perturbation and keep functioning, even if in a less efficient fashion.
Chaos Engineering is the practice of injecting perturbations in a production system and studying its resilience. It’s a scientific approach and it’s not testing! Testing should be done at unit, functional, and integration levels before anything is put in production. On the other end, Chaos Engineering lets you evaluate you system’s resilience in a real production setting, with all its functional partners available, at all levels of your architecture and deployment; it also lets you evaluate the resilience of the people in your teams, which is equally important to a service survival as the system itself.
At Scality, we’re familiar to deployments in environments where failure is not an option: some of our customers are Tier 1 providers, global banks, government agencies, global booking systems… they cannot afford to have their service down at all. Our architects designed the RING and Zenko to be resilient to chaotic situations. While the RING leverages the chord algorithm, private cloud replication, and shared nfs/object interfaces over a single storage, Zenko uses different mechanisms.
Zenko is designed with auto-scaling, self-healing, infrastructure checks and data availability in mind:
Auto-scaling: to handle sudden surges of traffic which are common at Tier 1 network providers, we chose Kubernetes for its native auto-scaling feature;
Self-healing: thanks to Kubernetes again, any pod that disappears will immediately be respawned elsewhere; together with microservices replica sets and auto-balancing of services on nodes, Zenko’s deployments are very resilient, even with several nodes gone;
Infrastructure checks: thanks to MetalK8s, our optimize K8s distribution, Zenko will not be deployed on a shaky network or an insufficient number of nodes, providing automatic protection from Network Chaos Monkey;
Data availability: the main purpose of Zenko is to give you multi-cloud control, so when one of its providers is down, an application can seamlessly failover to another using the S3 API; a single namespace over all clouds, with aggregated metadata search and replication workflows across all of them can, when used wisely, provide unlimited service availability.
All of that allows Zenko to pass the Chaos Engineering test: its components probe the hardware, scale up or down automatically, self-heal, offer stats on the system’s health via Orbit management UI. We love the quote attributed to Nora Jones (Senior Software Engineer @Netflix) about Chaos Engineering: Introducing Chaos is not the best way to meet your new colleagues, though it is the fastest: are your systems Chaos-proof? Come tell us in Zenko’s community… (fast)!
Wrapping up a great week at the OpenStack Summit in Vancouver where Zenko sponsored of the Superuser Awards. We had the honor to hand the trophy to the Ontario Institute for Cancer Research (OICR), a group of scientists that uses OpenStack to build the Cancer Genome Collaboratory: a cloud that enables research on the world’s largest and most comprehensive cancer genome dataset.
We like to see technology used to really improve the world by solving very difficult problems. The Superuser Award couldn’t be better placed, in the hands of small teams with highly constrained resources providing inspiration, amusement and wonder thanks to open source.
We hear inspiring stories from open source users all the time and that’s why we loved being with OpenStack: we’d love to see Zenko’s multi-cloud and replication capabilities added to the toolboxes of developers fixing the hard problems of the world.
Outside of the keynotes, the Zenko team spent time in Vancouver going to sessions, meeting the press and getting to know the ever changing OpenStack community. For me, this was the 13th OpenStack Summit (my first was in Boston, 2011). Years after, I found a strong group of people highly motivated to put OpenStack at the center of the infrastructure world. The announcements of the Kata Containers project and the spin-off of Zuul as its own project outside of OpenStack-infra team mark a pivotal moment for the Foundation. The comments we’ve heard on the Marketplace seemed all positive which seems to confirm the maturity of a community that has navigated across the hype cycle.
If you want to see extreme use cases of huge storage and distribution needs you go check the demos at NAB. Our colleagues at Scality are going to demonstrate how to use the multi-cloud capabilities of Zenko in a typical workflow, culminating in a multi-CDN distribution of a video. The demo consists in a movie being edited with Adobe Premiere to add some background music and overlays. Once saved, a simple Python script is invoked to transcode the movie in multiple versions optimized for different types of devices using FFMpeg. Each resulting file is uploaded to a single Zenko instance where the magic happens: the newly uploaded files are automatically replicated to two different locations, one in Amazon S3 and one in Azure Blob Storage, each with a CDN frontend.
Looking at the Python code that does the upload in Zenko you’ll notice how simple it is: just a RESTful PUT call to what looks just like a S3-compatible bucket. That’s what Zenko does: it keeps your code very simple and abstracts the complexity of managing a multi-cloud environment. The replication rules can be either managed with the Orbit web GUI or by editing configuration files. Besides, Zenko won’t mangle the files uploaded in the different clouds: we commit to giving you freedom and control of your data so you can still access the objects as they are uploaded, same file names and attributes.
Watch a recording of the demo below and if you’re at NAB, head to booth #SL9324 to meet Scality team.
L’équipe E2R5 (c’est la localisation « physique » du groupe au sein de l’école 42, le nom de la rangée dans les clusters de 300 postes où ils travaillent) regroupe 4 garçons (pas dans le vent) dont le projet (« votre mission, si vous l’acceptez… ») est de créer une sorte de micro-service pour Zenko. Plus précisément, ils doivent écrire un module pour rendre payantes (en cryptomonnaie) les actions qu’on peut réaliser via Zenko (lecture et écriture de données). Le fait de les tracer dans la blockchain permet par ailleurs d’augmenter la traçabilité des opérations, donc la confiance acheteur-vendeur.
Ce projet présente de nombreux avantages pour Guillaume, François, Cédric et Brice : il touche à la blockchain (le sujet dont tout le monde parle en ce moment), il leur permet de découvrir le javascript et de travailler sur le Web. Ils soulignent aussi que le format de ce hackathon les a séduit : il dure 5 jours au lieu de 2 habituellement et il doit déboucher sur du concret, en l’occurrence un prototype (« il va falloir pondre du code ») !
Ayant déjà eu l’occasion de participer ensemble à des événements de ce type, et issus de la même Piscine, ils n’ont pas de difficultés à travailler ensemble. Ils avancent pas à pas, en synergie malgré des tempéraments différents : Guillaume le « passe partout », François le « marrant » de la bande, Cédric le « néo » du groupe et Brice le « sorcier ».
Guillaume a passé un Bac STI en électronique, puis enchaîné sur un DUT en informatique. Après une rapide tentative en sciences cognitives, il a travaillé quelques années dans le secteur du bâtiment pour finalement rejoindre l’école 42, curieux de voir de quoi il s’agissait et avant de constater qu’on pouvait y rencontrer des gens venant d’horizons très différents.
François a passé le même Bac que Guillaume et a aussi évolué dans le secteur du bâtiment pendant quelques temps. Puis un jour, l’occasion d’aller en Australie avec un ami se présente. Il part ! Il garde un très bon souvenir de cette époque, même si il a apprécié de rentrer en France après 6 mois d’absence. Alors réparateur de portables et de tablettes, il découvre l’école 42 sur le net. Il essaye, c’est la révélation ! « Le code c’est la vie ! ».
Après avoir obtenu son Bac S, Cédric se lance dans études en maths et informatique ; puis il passe par l’alternance, touche à l’informatique de gestion, devient intérimaire quelques temps avant de partir un an en Australie et en Asie du Sud-Est. A son retour, sans idée précise de ce qu’il veut faire (toujours pas, précise t-il), peu séduit par les formations courtes en informatique qu’il trouve, il participe à la Piscine organisée en novembre 2016. Il n’a aujourd’hui aucun regret, au contraire.
Dernier « élu » du groupe, Brice. C’est celui qui a la plus grande culture informatique (notamment concernant l’open source). Il a un peu galéré après ses études avant de passer les tests de l’école 42 sur les conseils d’un ami et d’être retenu.
Aujourd’hui, ils constatent que le plus fou, depuis qu’ils sont à l’école 42, c’est que paradoxalement, ils n’ont jamais eu autant de liberté et autant travaillé.
Fin de l’histoire: E2R5 a sorti un prototype fonctionnel en cinq jours, malgré la foultitude de technologies qu’ils ont eu à apprendre! Le jury a retenu la créativité de leur projet, mais ne les a pas classé dans le top 3. Cependant, en ayant réussi à instancier un SmartContract et à “hacker” le code de CloudServer pour y ajouter le faire communiquer avec ce contrat, ils se sont prouvé qu’ils étaient capables de beaucoup!
