Narender Yadav / November 5, 2018
Cloud Storage, while established, has some challenges. It is seen as expensive and enterprises often need to plan and buy excess storage in advance. There can be latency involved in getting data, depending on the location of the cloud service provider.
To address some of these challenges, Enterprises have been looking at adopting Blockchain technology to create distributed storage. Let’s understand more about the applicability, the challenges, and the benefits.
However, other aspects of blockchain, like its distributed approach and security of data can be leveraged to form new kinds of storage models.
Blockchain is decentralized in nature. It consists of nodes or blocks of encrypted information and uses a distributed consensus model. Each node has a copy of the same data and transaction history and has the same rights. All information is secured cryptographically. Nodes in a peer-to-peer network make a distributed ledger system. Nodes are independent of each other. If one node leaves the network, the others can still work. There are no intermediaries or third parties needed for authentication.
Decentralized storage combines the best features of Blockchain technology and at the same time meets the practical demands of storing high volumes of data.
When you store files on Google Drive or an enterprise puts its data on Amazon S3, the data is on a centralized server somewhere, making it vulnerable to attacks and expensive to store.
This kind of cloud storage can be replaced by a decentralized data-storage model, which leverages Blockchain technology. In a decentralized storage, data is no longer on a single server. It is distributed across a network of nodes, similar to the distributed ledger of Blockchain, with many similar features.
In a distributed store, a file when added is broken into small pieces, encrypted, and sent to hard-drives (nodes) located all over the world, in a process called as sharding. Each shard is a part of the original and only the owner can access all the shards. The files are encrypted so no other node can read it.
Sia, Storj and Ethereum Swarm are companies that have begun using this technology to create peer-to-peer enterprise storage solutions. This is a very efficient model in which the storage scales very easily and becomes cost effective.
If enterprises buy storage from a few dedicated providers, the providers decide the cost and it is not very competitive. Distributed storage is cheaper than server-based storage or cloud storage. There is no expensive server farm as in cloud storage. Individual nodes in the network offer up the extra space on their servers or devices to the distributed ledger storage. Customers looking for data-storage can buy from them, rather than the monolithic cloud vendors. This can make storage up to a tenth as cheap as that from players like AWS.
When large volumes of data are involved, syncing it over the storage chain of nodes can be very slow. If data is coming from real-time IoT devices, this situation is hardly ideal.
After data is stored collectively as shards over the nodes, a technique known as swarming is used in a distributed storage system, to speed up data transfer. This uses the same principle as torrents. Devices in a large group, or swarm, in a peer-to-peer network transfer data to and from the nearest nodes in parallel. This reduces latency.
An interesting solution now emerging is the marketplace. You can leverage a platform like Ethereum for such distributed storage and add features of ownership and contracts and create a storage marketplace which will be very cost effective.
Imagine a peer-to-peer storage marketplace (enterprise or individual). It is a community of storage providers and consumers in a network or grid. The providers sell their unused storage and buyers purchase it to upload their data. Files are broken into shards and distributed across multiple nodes spread geographically.
The marketplace can identify unused resources from hosts and match them with large demand from consumers. When your storage needs go up, you buy more storage and relinquish it when no longer needed. Data is stored redundantly. If anyone needs to reclaim their storage capacity, the marketplace ensures this by moving data to another appropriate location, if that location also meets the SLAs and performance criteria. The buyers can buy storage from a nearby location rather than from a provider across the globe.
In this marketplace, users can monetize their unused disk space. Pricing would be very competitive due to a large number of sellers. Costs would be low also because the marketplace is selling capacity that is already spare. No one needs to buy special infrastructure.
The marketplace ensures that a contract exists between the storage owner and owner of the data and the data has been written to storage. It tracks all buying and selling and manages payments effectively for both parties. Similar to a cryptocurrency ledger of transactions, the storage marketplace creates records of data movement.
Purchase and sale transactions are immutable records, and the marketplace is trustworthy. There is no need for a third party for authentication. Such a distributed storage marketplace can fundamentally transform the enterprise storage scenario.
Blockchain is slated to make an impact on lives and business in ways as fundamental as the internet itself. From what is visible even now, it seems clear that it will have a pretty transformational impact on the world of data-storage too.