Speed up deployments

[iperdomo]

The current approach to make deployments is a serial process. Going one by one and calling ant update. This process makes unnecessary work as the only change between instances is appengine-web.xml. It seems that deploying all active instances take ~8h which is too much.

We need to:

• Avoid unnecessary work
• Investigate the possibility of using a pull based approach instead of pushing the release to each instance.

[iperdomo]

To avoid unnecessary work the deployment process will build the Dashboard (JS) and compile the application once, then change only the config file appengine-web.xml per instance

[iperdomo]

Related to a pull based approach, we have successfully deployed a Flow instance using the Google App Engine Admin API

The idea is that we can create a new version (or update an existing one) by making a POST request to the admin API with a Version request.

The Version request is a JSON payload that contains the list of files and a pointer fo a Google Storage bucket url from where the instance and pull the files for deploying a new version.

[iperdomo]

There are some unknowns on how to translate the <security-constraints> to handlers as the app.yaml or app.json requires. So the strategy for deployments will be the following:

• Deploy the application using gcloud to a known and default instance (e.g. akvoflowsandbox)
  • The gcloud app deploy uses a Google Storage bucket to upload the files that will be the source of the deployment
• After a successful deployment of version 1 and dataprocessor we use the Google App Engine Admin API to GET the current definition of the version of the recently deployed known instance. This will produce a JSON payload similar to the one we need for making POST requests.
• We take that JSON definition of the version and change the link to the appengine-web.xml (the only file that changes) and make a POST to update a version to another known instance (e.g. uat1)
• We verify that the new deployment went OK by making a couple requests to known URLs
• After verification, we loop through a list of instances to deploy making the same POST request with the only change of the link to appengine-web.xml
  • This can be a serial POST but the caller doesn't need to wait for completion. We get a operation id that we can check few minutes later for status (and keep polling until is done)

[muloem]

@iperdomo There is a small issue with the script. Even when the GH_USER and token environment variables are set, I still get a prompt to enter them.