Recently, we were engaged by one of our clients to investigate improvements to their existing AppDynamics configuration in order to provide greater insight in to specific areas of their application and also to provide more granular alerting capabilities. The result of this work was the creation of 11 health rules, specific to an application tier, which provided the granularity that was desired.
We then had a problem. How to deploy these health rules and the dashboard to each of the 14 application tiers in a timely and automated fashion? Secondary to this was a desire to define a standard set of Health Rules that could be easily applied to future projects as and when required.
The solution came by leveraging the AppDynamics REST API which provided a quick and simple way of exporting, and importing, health rules in bulk. Initially, the new Health Rules were entered manually for a single Application Tier and then exported using the API. The result of this operation is an XML file that described the configured rules.
It was quickly identified that the health rules defined in the XML document could be easily templated by obtaining information such as the name of the application tier and its numeric ID. A quick search and replace through the XML document resulted in a template document containing specific keywords in place of the defined tier names and IDs. A simple shell script was then used to modify this template and output rulesets for any application tier we desired, based on an input file containing a list of the application tier names and IDs that the rules were to be applied to.
Import of the resulting XML document via the API was initially performed via a REST API client which allowed the rules to be imported on a per-application basis. The result was the creation of 154 health rules in under an hour, providing a huge saving on time and effort. The API provides the ability to optionally overwrite existing rules, based on the rule name, so modifications and tweaks could be performed easily and quickly as required.
Once the health rules had been templated, the next logical step was to also template the custom dashboards we had designed. The process for this was similar to the health rules. A dashboard that had been created manually was exported via the API and templated in the same way as the health rules, replicated for each appropriate application tier, and imported via the API. This ensured each dashboard had the same look and feel and that the data being presented was consistent across all application tiers.
The completed health rules and dashboards are also providing our client with an unparalleled insight in to the performance of their application and were actively used by engineers and developers alike during a large migration and go-live event. Performance bottlenecks in each application tier were quickly identified and rectified before they became a problem. A big win for us and our client.
Our work to template the creation of health rules and dashboards ensures each and every health rule and dashboard can now be defined in a standard way with no variations or misconfigurations that can be introduced in a purely manual process. The next step is to hold the templates in source control and use a CI solution, such as Jenkins, to generate and import the rules. This would also allow any changes to the base rule sets or dashboard designs to be easily and automatically rolled out.
By Wayne, a DevOpsGuy