Cloud Crawling – How tomorrow’s cloud native application may operate (Part 2)

This article is continuation of Part 1. The cloud crawling solution provides a novel method of realizing a cloud-based architecture in which an enterprise can utilize clouds from different vendors across the world to locate its application, data and assets closer to the end user. In this article, I present a conceptual solution for it.

Cloud Crawling Solution

The solution comprises of four separate concerns:

  1. Cloud discovery
  2. Cloud selection
  3. Term negotiation
  4. Cloud orchestration
Cloud discovery

As the number of cloud vendors grows, it isn’t a stretch to assume the availability of a third party cloud discovery services in the future. These services will provide a directory lookup for different cloud vendors, very similar to the Domain Name Service. In this case, the service maintains an updated list of all the clouds for different regions and when queried by a region it returns a list of end points for all the clouds serving that region. The end points are addresses of cloud agents. A cloud agent represents the entry point for a cloud. It helps negotiate terms and enable deployment on the cloud.

The cloud discovery process finds all the available clouds for a given region by querying a cloud discovery service.

The following sequence diagram demonstrates the cloud discovery mechanism,

Cloud selection

The Cloud selection process selects a cloud from the list of clouds found by the cloud discovery process for a given region. The cloud that is selected will be the one on to which the application will then deploy itself.

The cloud selection process may compare the different cloud options using certain metrics such as pricing, available capacity, QoS statistics, past usage experience, SLAs, PCI compliance, number of PoPs etc.

The following sequence diagram demonstrates the cloud selection mechanism,

Term negotiation

The Term negotiation process kicks in after the cloud selection process has selected a cloud vendor. As part of the term negotiation process, the application and the cloud agent exchange their terms of use and accept or reject the contract. The contract may be binding on the cloud vendor to ensure the desired QoS for the application.

Cloud orchestration

Cloud orchestration acts as a supervisor and manages all the other cloud crawling processes – discovery, selection and term negotiation. Orchestration can be demand based or schedule based.

The cloud orchestration and all the processes which it manages make up the cloud crawling module of the application.

Example Application –

Demand based orchestration

In this approach, if the application discovers that most of the requests are coming from a particular region then the cloud orchestration module will initiate a crawl to a cloud in that region to move closer to the user base and better serve the demand.

e.g. In the figure below, the application is first brought up or seeded on a cloud in Region x. It then starts receiving request from several clients in Region y (1). As a response to the demand from Region y, the cloud orchestration module migrates the complete application to a cloud in Region y – cloud A (2). The clients in Region y are now served through cloud A in the same region thus lowering network latency (3). After some time, the traffic monitoring component registers several requests from clients in Region z (4). Again, the same steps are repeated and the application is migrated to cloud B, a cloud in Region z (5). The requests from clients in region z are now served by the deployment in cloud B (6).

Time based orchestration

In this approach the cloud orchestration initiates the crawling process as per a predefined schedule.

e.g. In the figure below, the application is first brought up in Region x and a schedule is defined as follows,

  • Time T0 – Region x
  • Time T1 – Region y
  • Time T2 – Region z

The orchestration module dutifully follows the schedule and deploys the application on Cloud A which is in Region y on time T1 and then cloud B which is in region z on time T2.


Multi-cloud deployment is already a reality and cloud crawling takes it one step further. I hope you enjoyed reading this article and would love to get your feedback.