Cloud Crawling – How tomorrow’s cloud native application may operate


With the proliferation of cloud technologies, computing resources, storage resources and even streaming resources have been almost reduced to a commodity which can be reserved, utilized and then relinquished in real-time without the need of exclusive ownership.

Currently a handful of cloud vendors such as Amazon, Microsoft and Google dominate the cloud computing landscape with their cloud offerings. However, regional clouds backed by mid-size companies are also gradually becoming significant. Regional clouds have a limited PoP and may be limited to a given region. However, they have the advantage of being flexible enough to conform to different domain specific restrictions that customers may demand [1]. Regional clouds, above everything, help make the cloud market-space more competitive.

In the future there will be far more choices of clouds for any organization to choose from. These choices will consist of a broad mix of regional and global clouds. Furthermore, there are already standards in the making which allow applications to communicate with any cloud vendor using a vendor-neutral or cloud agnostic protocol [2].


This myriad of cloud choices across the globe combined with the idea of dynamic reservation of resources [3 & 4] could let an application “follow its users” across regions and countries without any manual Dev-ops intervention. An application will be able to itself negotiate with different cloud vendors, reserve resources and migrate-in and migrate-out as needed. This is what I term as – Cloud crawling, aprocess of dynamic resource migration across different clouds on-the-fly.

Lets look at an example. Imagine a news service provider which provides daily world news and videos. The service is completely hosted on the cloud. Let say the service is initially seeded on a cloud geographically located on the west coast. As viewers from different continents and countries consume the news service, the service on the fly and without any manual intervention negotiates with different cloud vendors located close to its viewers. It deploys itself on those clouds by standing up processing instances and by ingesting assets on the clouds CDNs. It also un-deploys itself from a cloud based on the traffic or maybe based on a fixed schedule. In this manner, the service can potentially migrate across the globe and appear to be ubiquitous to all its users while serving content with minimal network latency.

The following diagram further illustrates this example of how the service which started on the US west coast over a period of time spawned across clouds and migrated across the world. At each point it discovered different regional, national and international clouds, selected the clouds based on certain criteria, negotiated service agreements and accomplished a new greenfield deployment. After the user load from any particular region decreased, it brought down its deployment and relinquished those cloud resources.

In essence, the application “crawls” across different clouds and regions. This builds upon the current technique of deploying the same application simultaneously on different clouds, known as cloud bursting. However, while cloud bursting is limited to providing redundancy, cloud crawling provides a truly dynamic deployment such that the application itself discovers and manages its own migration to different clouds.


Cloud crawling represents a truly cloud-agnostic and dynamic solution for cloud native applications of tomorrow. It helps avoid vendor lock-in and exploits the competition between cloud providers on an on-going basis.

If you found this article interesting, I invite you to Part-2 where I describe a high-level design of how Cloud Crawling can perhaps be implemented.


1.    The “Regional” Cloud: A Case Study


3.      Patent – US20110145393 – Method for dynamic reservation of cloud and on premises computing resources for software execution (

4.      Patent – US20100076856A1 -Real-time auction of cloud computing resources (