End to End development can be frustrating unless the combination of the right tools is used. Many of you are aware of End to End development definition which includes complete development up to and including application in production. Where one can have released software into production as and when desired possibly twice a day.
How it makes it efficient and possible?
Let me explain two ways where End to End development was done in the past.
1. Using Homegrown deployment
A financial service where 3rd Party software (PAAS) Cloud Foundry or OpenShift was not allowed. Complete End to End development was done using bespoke software deploying to replicated Linux boxes in three different time zones EMEA, Asia Specific and USA. You can see the overhead of deploying into three different time zones. If a deployment fails and one needs to roll back ends up total chaos and disaster results.
Bank wanted to deploy into production 4 to 6 times and split the teams into small pods where a pod can have up to 2 or 3 well experienced full-stack developers who can work on entire End to End seamlessly. Deployment is done using normal technology using IntelliJ for code development, Maven for Build and Jenkins as part of continuous integration environment.
The critical path was the deployment plan:
1.1 Commit Shell Script
1.2 Rollback Script
1.3 Instruction for business to sign off (Please note time zones)
In a nutshell, spending more time in deploying onto production server than development. The whole system was prone to errors in case any dependencies are missing. Everything needs to be perfect and all micro-services needed to be on Linux replicated servers.
2. Using Homegrown deployment using CF commands
Same financial services but with different team wanted to use Micro-services within their team. End to End concept for build, test and deliver using Jenkins Server to System environment. Here within Jenkins server shell scripting using CF command line syntax was adopted to stage and push to PWS.
The whole concept was smooth except CF commands were handcrafted within shell scripting. There was a number of disadvantages as follows:
1. Security integrity was compromised with the script.
2. No fall-back error handling
3. Exceptions in script frequently needed to write more code in scripts! disaster
4. Homegrown error handling
5. No Post-deployment check and rollback
6. Hand coding of blue-green deployment
7. Had to aggregate 2 logs PCF and Jenkins
8. A lot of effort needed for blue/green deployment to make it green
9. Binaries in Nexus another level of discomfort for managing archives and latest builds
10. Simple microservice taking well over 10 minutes and building entire infrastructure taking over two hours.
The entire deployment was handed over to the operational team. Where development team will build and test and use scripts or jobs generated by the operational team.
Operation or deployment parameters kept separately. Jenkins parameters kept with Jenkins server and development parameters kept with the development.
Time constraint taking a few hours to build the entire project.
This is where using development and deployment has the advantage as a pure End to End developer development, Build, test and deployment build in within the same structure. DevOps having full knowledge development and operation and NO segregation.
3. This is using the right way to build and test similar to (1) and (2) but deployment using concourse.
When using Concourse CI & Cloud Foundry deploying, to PWS makes life easier. Keeps development and deployment configuration at one place.
Because we were doing concourse, we only had to the YAML task once, and then each of our future code changes did not require any more work on the deployment configuration.
In order to make it work, we can simply replicate the same YAML but with a different configuration (resource change) to deploy to multiple regions.
A major advantage was that because we were deploying to 3 different regions, could only deploy at the close of the business day. Which means deploys happen at 10 am for the Asia Pacific, 6 pm for EMEA, and around 10 pm for the USA. But with concourse, we can deploy anytime due to zero downtime deployment (providing business approval :)
Conclusion
1. Using concourse or GitLab — deploy within minutes
2. No pain/No large scripts
3. Done in minutes not days/weeks
4. No need to master CF
5. Easy integration with Concourse/CF
With me well over 15 years of experience, it was a pleasure and sheer happiness to see how effective and how efficiently I manage to deploy using concourse painlessly with little or no training.
All the configuration and deployment under one hood GIT repository. Easier to manage as being in one place and easier to deploy from the same place.
The concept is employed by the majority of the banks including Santander and I feel proud to be part of it.
Some of the products supplied by Business Integration Software Ltd
Online Timesheet Software
Multiple Choice Question Software
