How we reduced SOA OSB provisioning from 4 days to 4 hours

Posted on by Ramesh Poonati

In our organization, we’ve undertaken the task of upgrading both SOA and OSB from version 12.2.1.3 to 12.2.1.4. This involves an out-of-place migration approach, wherein we create a new environment and switch traffic during the cutover. As you know, this process typically requires several days we opt for manual provisioing.

A very significant amount of automation work was previously accomplished for 12.2.1.3 by our predecessors. Despite these efforts, the process still used to take lot of time due to following factors:

  • Bugs in existing automation. For example, SOA composite deployments is not stable which warrants lot of manual work
  • Import of 10s of TLS Certificates into KSS
  • Manual retiring and shutting down of services
  • Monitoring setup
  • Miscellaneous: Increasing Log file count, enabling GC logging, changing heap parameters, bash profile changes

While our current automation initiatives have yielded substantial savings, we aimed to capitalize on this upgrade activity to enable end-to-end automation, starting from from VM provisioning to the setup of monitoring. This extended automation supports both upgrade processes and the creation of new instances. For instance, during upgrades, it deploys services by exporting from the prior version and subsequently deploying them onto the upgraded domain. In cases where brand new environments are required, the automation exports services from the reference environment and deploys on the new environment. To achieve this, we leveraged the following technology stack:

  • VRA (Used by Linux Team for VM provisioning)
  • Ansible
  • WLST/Python Scripts
  • Bash Scripts

At a high level the tasks involved are:

  • Copy product binaries from reference environment and run RCU
  • Unzip Java and rename folder to generic name without version so that subsequent Java upgrades are painless
  • Create domain
  • Use domain override files to configure GC logging, heap parameters, heap dump, disable derby DB. This keeps custom changes to domain config files in one place and hence it is easy to manage.
  • Create OHS domain and setup OHS configuration
  • Pack and unpack domain
  • Node manager enrolling
  • Import TLS/SSL certificates
  • DB, SOA adapter configuration
  • Applying latest patches
  • Datasource and JMS artifacts creation
  • CSF Key creation, email configuration, datasources config changes like XA configuration, JTA Timeout, Deploy custom JAVA EE applications, increase log file count
  • Deploy SOA/OSB services
  • Shutdown and Retire services similar to old env
  • Add aliases to bash profile to save keystrokes while switching directories. For example, if we type adomain, it will take us to admin server domain directory. Sample bash profiles are available here.
  • Enable key based authentication between nodes
  • Configure monitoring scripts according to env and setup cron jobs
  • Install jolokia and Telegraf agents and send metrics to Wavefront for visualization and alerting

Thanks a ton Aswin for continuously helping me in this upgrade endeavor.

Posted In Uncategorised

Related Post