Error Report Using Mulesoft

Note: While copy/paste the code snippet from blog please covert quotes and double quotes.

Usually every middleware has some or other kind of alter notification implementation for error scenarios. We are trying to explore the option of generating Error Report to combine all errors occurred using MuleSoft.

Objective of this blog to create a generic component which can be reused across all projects within Mulesoft.

MuleSoft Options to Notify Error Notification?

• CloudHub Notification and CloudHub Alter
• Using Send EMail activity

Drawbacks:

• Limited number of Alters can be configured
• Individual mail will be sent for each alert which will cause multiple mails in Inbox

Overview of Error Reporting:

• Whenever error occur it will create an error object and capture the key identifier for record and along with error message
• Error Object will be stored in Object Store (Object Store can be replaced by other persistent storage)
• A scheduler will run on specified time interval and accumulate all the error and send the report as attachment

Benefits:

• Reduced number of mails
• Accumulated Report for the errors
• Can be used to generate report and present to different team
• Run Application

Sample Mule Flow:

This flow to showcase uses of the error reporting component.

• Scheduler will run every minute
• Payload will be set as an Array with value from 1 to 100
• Batch JOB Process data in batches and in parallel
• Generate error will throw error for every odd record
• Accumulate error will set the accumulate_error variable

[
 {
 “objectType”: “ObjectType”,
 “keys”: [
 {
 “name”: “Timestamp”,
 “value”: “2020–07–08T14:44:38.695Z”
 },
 {
 “name”: “Flow Name”,
 “value”: “FlowName”
 },
 {
 “name”: “Component Name”,
 “value”: “ComponentName”
 },
 {
 “name”: “Primary Key”,
 “value”: “PrimaryKey”
 }
 ],
 “error”: {
 “description”: “Error Description”
 }
 }
]

• ObjectType, FlowName, ComponentName, PrimaryKey can be changed based on need also more name value elements can be added based on requirement
• Notify Error will aggregate all the error Objects and generate error Report

Batch Error Report Flow

<?xml version=”1.0" encoding=”UTF-8"?>
<mule xmlns:ee=”http://www.mulesoft.org/schema/mule/ee/core" xmlns:batch=”http://www.mulesoft.org/schema/mule/batch"
 xmlns=”http://www.mulesoft.org/schema/mule/core"
 xmlns:doc=”http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=”http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/batch http://www.mulesoft.org/schema/mule/batch/current/mule-batch.xsd
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd">
 <flow name=”batch-error-reportFlow” doc:id=”7f68b313–86b1–4e28–9e4d-1f90a6b1b283" >
 <scheduler doc:name=”Scheduler” doc:id=”91408ed3-a06c-4187-af37–576f321d9df7" >
 <scheduling-strategy >
 <fixed-frequency frequency=”1" timeUnit=”MINUTES”/>
 </scheduling-strategy>
 </scheduler>
 <ee:transform doc:name=”payload” doc:id=”248821f4–81fd-42f3–940c-a1b743afcfd4" >
 <ee:message >
 <ee:set-payload ><![CDATA[%dw 2.0
output application/java
 — -
1 to 100]]></ee:set-payload>
 </ee:message>
 </ee:transform>
 <batch:job jobName=”batch-error-reportBatch_Job” doc:id=”45260843–3555–4d6a-81ee-ae2f263599fc” >
 <batch:process-records >
 <batch:step name=”Batch_Step” doc:id=”fc5904ae-7726–4b89-a8a8-e78a6d5b147b” >
 <try doc:name=”Try” doc:id=”5df25f3d-c5bf-4ff8-bae6-a2f822de0ffa” >
 <ee:transform doc:name=”generateError” doc:id=”942677b6–050a-4e90–950e-9ec888729a99" >
 <ee:message >
 <ee:set-payload ><![CDATA[%dw 2.0
output application/java
 — -
if ((payload mod 2) == 0)
 payload * 2
else
 payload / 0]]></ee:set-payload>
 </ee:message>
 </ee:transform>
 <error-handler >
 <on-error-continue enableNotifications=”true” logException=”true” doc:name=”On Error Continue” doc:id=”595d45a3–021c-40b5-a5e7-be8fb7580f0e” type=”ANY” >
 <ee:transform doc:name=”accumulateError” doc:id=”0e9af43f-70c3–40d8–9b05-f6bd6ce6ab91" >
 <ee:message />
 <ee:variables >
 <ee:set-variable variableName=”accumulate_error” ><![CDATA[%dw 2.0
output application/json
 — -
[{
 “objectType”: “ObjectType”,
 “keys”:[
 {
 “name”: “Timestamp”,
 “value”: now()
 }, {
 “name”: “Flow Name”,
 “value”: “batch-error-reportFlow”
 }, {
 “name”: “Component Name”,
 “value”: “Batch JoB”
 },{
 “name”: “Primary Key”,
 “value”: payload 
 }
 ],
 “error”:{
 “description”: write(error,’application/json’)
}
}]]]></ee:set-variable>
 </ee:variables>
 </ee:transform>
 <flow-ref doc:name=”accumulate-error-os” doc:id=”caf43a4a-2829–4c2c-8bb4-c90db4b03d08" name=”accumulate-error-os” />
 </on-error-continue>
 </error-handler>
 </try>
 </batch:step>
 </batch:process-records>
 <batch:on-complete >
 <flow-ref doc:name=”notify-errors” doc:id=”17dfb5db-8a04–4dd1-b8cf-2ec43fe5eab3" name=”notify-errors”/>
 </batch:on-complete>
 </batch:job>
 </flow>
</mule>

Error Reporting Flow:

• Scheduler process will run on specified internal like every one hour
• It will get all the data from Object Store (Persistent Store)
• Combine data based on Object Type
• Send mail and attach report

Accumulate Error OS

Notify Error

<?xml version=”1.0" encoding=”UTF-8"?>
<mule xmlns:os=”http://www.mulesoft.org/schema/mule/os"
 xmlns:email=”http://www.mulesoft.org/schema/mule/email"
 xmlns:db=”http://www.mulesoft.org/schema/mule/db"
 xmlns:json=”http://www.mulesoft.org/schema/mule/json"
 xmlns:http=”http://www.mulesoft.org/schema/mule/http"
 xmlns:cloudhub=”http://www.mulesoft.org/schema/mule/cloudhub"
 xmlns:scripting=”http://www.mulesoft.org/schema/mule/scripting" xmlns:ee=”http://www.mulesoft.org/schema/mule/ee/core"
 xmlns:file=”http://www.mulesoft.org/schema/mule/file"
 xmlns:jms=”http://www.mulesoft.org/schema/mule/jms"
 xmlns:apikit=”http://www.mulesoft.org/schema/mule/apikit"
 xmlns:context=”http://www.springframework.org/schema/context" xmlns:tracking=”http://www.mulesoft.org/schema/mule/ee/tracking"
 xmlns:metadata=”http://www.mulesoft.org/schema/mule/metadata"
 xmlns=”http://www.mulesoft.org/schema/mule/core" xmlns:doc=”http://www.mulesoft.org/schema/mule/documentation"
 xmlns:spring=”http://www.springframework.org/schema/beans" xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=”
http://www.mulesoft.org/schema/mule/email http://www.mulesoft.org/schema/mule/email/current/mule-email.xsd 
http://www.mulesoft.org/schema/mule/db http://www.mulesoft.org/schema/mule/db/current/mule-db.xsd 
http://www.mulesoft.org/schema/mule/json http://www.mulesoft.org/schema/mule/json/current/mule-json.xsd 
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd 
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/ee/tracking http://www.mulesoft.org/schema/mule/ee/tracking/current/mule-tracking-ee.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-current.xsd
http://www.mulesoft.org/schema/mule/apikit http://www.mulesoft.org/schema/mule/apikit/current/mule-apikit.xsd
http://www.mulesoft.org/schema/mule/jms http://www.mulesoft.org/schema/mule/jms/current/mule-jms.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.mulesoft.org/schema/mule/scripting http://www.mulesoft.org/schema/mule/scripting/current/mule-scripting.xsd
http://www.mulesoft.org/schema/mule/cloudhub http://www.mulesoft.org/schema/mule/cloudhub/current/mule-cloudhub.xsd
http://www.mulesoft.org/schema/mule/os http://www.mulesoft.org/schema/mule/os/current/mule-os.xsd">
 <! — <email:smtp-config name=”Email_SMTP” doc:name=”Email SMTP” doc:id=”1db316ac-7f26–4647-ac7d-821f8e363eb3" >
 <email:smtp-connection host=”smtp.gmail.com” port=”465" user=”siddhesh.desai@apisero.com” password=”Sidbb@8520" />
 </email:smtp-config> →
 <sub-flow name=”accumulate-error-os” doc:id=”a2560edc-bf0b-4991–9f99-a43d678a6959" >
 <try doc:name=”Try” doc:id=”11b9a9e6-dd4c-4558–98b3-fa1aba03e69b” >
 <os:store doc:name=”store error obj” doc:id=”599bf90d-23ed-4cb4–8dd5–33d61e467169" key=’#[“accumulate_error_” ++ uuid()]’ failOnNullValue=”false” objectStore=”Object_store”>
 <os:value><![CDATA[#[output application/java
 — -
“$(write(vars.accumulate_error, “application/json”))”]]]></os:value>
 </os:store>
 <error-handler >
 <on-error-propagate enableNotifications=”true” logException=”true” doc:name=”On Error Propagate” doc:id=”c2af34bd-c4d3–43d8–87af-0de96ff4697c” >
 <logger level=”ERROR” doc:name=”Logger” doc:id=”ef5d2e51–9a3a-4768-ad8b-e07eb09945c6" message=”Unable to store data in Object Store”/>
 </on-error-propagate>
 </error-handler>
 </try>
 </sub-flow>
 <flow name=”notify-errors” doc:id=”c16176c0-be34–4826-af0e-1378a58e9cc8" initialState=”started”>
 <! — <scheduler doc:name=”Scheduler: Fetch Error Info” doc:id=”22a5436b-920c-4ae6–997b-53bab8911efe” >
 <scheduling-strategy >
 <fixed-frequency frequency=”${scheduler.frequency}” timeUnit=”${scheduler.timeUnit}” />
 </scheduling-strategy> 
 </scheduler> →
 <! — <db:select doc:name=”Retrieve Object and Emails” doc:id=”3f0a103b-8608–4f14–95c3-e7a5f632f4b0" config-ref=”Database_Config” target=”muleEmails” transactionalAction=”NOT_SUPPORTED”>
 <db:sql>SELECT OBJECT, EMAILADDRESS from MULEEMAILS</db:sql>
 </db:select> →
 <scheduler doc:name=”Scheduler” doc:id=”41d205a0–9a9b-4fb0–8030–4a912b636794" >
 <scheduling-strategy >
 <fixed-frequency frequency=”3" timeUnit=”MINUTES”/>
 </scheduling-strategy>
 </scheduler>
 <os:retrieve-all doc:name=”get all error objects” doc:id=”9e09e34b-065f-4625-b0c8-a7812624e222" targetValue=’#[%dw 2.0
output application/json
 — -
flatten((payload..) map ((item, index) -&gt;
 read (item, “application/json”)
)) groupBy ((item, index) -&gt; item.objectType )]’ objectStore=”Object_store” target=”retrievedErrors”/>
 <choice doc:name=”Choice” doc:id=”33a0d64d-50cb-4b35–8d41–6513a74096eb”>
 <when expression=”#[sizeOf(vars.retrievedErrors) &gt; 0]”>
 <foreach doc:name=”For Each” doc:id=”4598b775–6077–4b08–8c76–3087f23b426a” collection=”#[vars.retrievedErrors]”>
 <ee:transform doc:name=”payload” doc:id=”f3bb60ba-98d7–4441–8259-cf5be9b38f0c”>
 <ee:message>
 <ee:set-payload><![CDATA[%dw 2.0
output application/csv
fun arrToObj(arr) = arr reduce ((env, obj={}) -> obj ++ env)
fun combineKeys(keys) = (arrToObj (keys map ((item, index) ->
 str: item.name ++ “: “ ++ item.value
)).*str joinBy “ | “)
fun nameValuePairToJson(keys) = (keys map ((item, index) ->
 “$(item.name)”: item.value
))
 — -
payload[0] map ((item, index) -> arrToObj(nameValuePairToJson(flatten(item.keys))) ++ {
 ErrorDesc: item.error.description
})]]></ee:set-payload>
 </ee:message>
 <ee:variables>
 <ee:set-variable variableName=”objectType”><![CDATA[%dw 2.0
output application/json
 — -
payload pluck ($$)]]></ee:set-variable>
 </ee:variables>
 </ee:transform>
 <set-variable value=’#[“ErrorReport_” ++ (now() as String {format: “yyyyMMddHHmmssSSS”}) ++ “.csv”]’ doc:name=”Attachment fileName” doc:id=”d3881afb-400c-4330–979b-75a7f4e3c6d7" variableName=”fileName” />
 <email:send doc:name=”Send Error Report via Email” doc:id=”4355b577–981c-429f-a666-a2a6640c2bcf” fromAddress=”${mail.from}” subject=”Error Report” config-ref=”Gmail_SMTP”>
 <email:to-addresses>
 <email:to-address value=”${mail.to}” />
 <email:to-address value=”${mail.to}” />
 </email:to-addresses>
 <email:body>
 <email:content><![CDATA[#[%dw 2.0
output text/plain
 — -
“Failed Records logs are added in attachments please refer same.”]]]></email:content>
 </email:body>
 <email:attachments><![CDATA[#[%dw 2.0
output text/csv
 — -
{
 ‘$(vars.fileName)’: payload
 }]]]></email:attachments>
 </email:send>
</foreach>
 </when>
 <otherwise>
 <logger level=”INFO” doc:name=”Logger” doc:id=”a302abc7–6af0–4f00–82b7–7a492fa84c37" message=”All Records are successfully processed.” />
 </otherwise>
 </choice>
 <os:clear doc:name=”Clear” doc:id=”21171d9e-1474–4206–9203–3a0695c0bcb0" objectStore=”Object_store” />
 <error-handler >
 <on-error-propagate enableNotifications=”true” logException=”true” doc:name=”On Error Propagate” doc:id=”3ec241cd-ae72–4096-add5–41fa1f0aeed1" >
 <logger level=”ERROR” doc:name=”Logger” doc:id=”6e026028–0381–4840-bd05-fb8d5edb8fa3" message=”Unable to notify error”/>
 </on-error-propagate>
 </error-handler>
</flow>
</mule>

Sample Error Report:

Sample Error Report