Logging using Elastic Cloud for MuleSoft using HTTP Appender

  • ELK is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana
  • Elasticsearch is a search and analytics engine
  • Elasticsearch is an open source, full-text search and analysis engine, based on the Apache Lucene search engine
  • Logstash is a server-side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a “stash” like Elasticsearch
  • Kibana lets users visualize data with charts and graphs in Elasticsearch
  • Log aggregation and efficient searching
  • Generic Search
  • It’s Interoperable
  • It’s Open Source
  • It’s Managed
  • On-demand computing
  • Pay only for what you use
  • Failover and fault tolerance
  • Common coding
  • Ease of implementation
https://elastic.co
  • When you login for first time It will ask you to create a default ELK deployment which will install ElasticSearch and Kibana installation as below
Deployed Instances
  • It will also create username and password for elastic search please note it down as it will be use full for later stages
  • Go to Project
  • Open src/main/resoruces/log4j2.xml
  • Add below xml tag in Configuration/Appenders
<Http name=”ELK-Cloud” url=”https://bf1718d968f54023b4b4e75badd4fee2.us-east-1.aws.found.io:9243/applogs/_doc"> 
<Property name=”Authorization” value=”Basic ZWxhc3RpYzoxREd6b0RTMWdTNTdNYlJMbjJ5S2UyU1k=” />
<Property name=”Content-Type” value=”application/json” />
<PatternLayout pattern=’%msg’ />
</Http>
  • url should follow pattern as metioned https://<Elastic-Server-Name>:<Port>/<Index-Name>/_doc
<AppenderRef ref=”ELK-Cloud” />
  • Run Application
  • See the Console Log
Console Output
Kibana Portal -> Management
  • Click on Create Index Pattern
  • You can see a new index applogs is created. Select it and click on Next Step
Kibana Portal Index Creation Step 1
  • Click on drop-down and select @timestamp and click on Create Index Pattern
Kibana Portal Index Creation Step 2
  • Start Mule application
  • Run few cases so Mule file can generate the logs
Console Output
Kibana Portal -> Discover
  • In Search you can write any suitable expression to search specific text from log file
MuleSoft Application Folder Structure
  • MuleSoft is not responsible for lost logging data due to misconfiguration of your own log4j appender.
  • MuleSoft is also not responsible for misconfigurations that result in performance degradation, running out of disk space, or other side effects.
  • When you disable the default CloudHub application logs, then only the system logs are available. For application worker logs, please check your own application’s logging system. Downloading logs is not an option in this scenario.
  • Only Asynchronous log appenders can be used, Synchronous appenders should not be used.
  • Use asynchronous loggers and not synchronous ones is to avoid threading issues. Synchronous loggers can lock threads waiting for responses.
<Http name=”ELK-Cloud” url=”https://bf1718d968f54023b4b4e75badd4fee2.us-east-1.aws.found.io:9243/applogs/_doc"> 
<Property name=”Authorization” value=”Basic ZWxhc3RpYzoxREd6b0RTMWdTNTdNYlJMbjJ5S2UyU1k=” />
<Property name=”Content-Type” value=”application/json” />
<PatternLayout pattern=’%msg’ />
</Http>
<Log4J2CloudhubLogAppender name=”CLOUDHUB”
addressProvider=”com.mulesoft.ch.logging.DefaultAggregatorAddressProvider”
applicationContext=”com.mulesoft.ch.logging.DefaultApplicationContext”
appendRetryIntervalMs=”${sys:logging.appendRetryInterval}”
appendMaxAttempts=”${sys:logging.appendMaxAttempts}”
batchSendIntervalMs=”${sys:logging.batchSendInterval}”
batchMaxRecords=”${sys:logging.batchMaxRecords}” memBufferMaxSize=”${sys:logging.memBufferMaxSize}”
journalMaxWriteBatchSize=”${sys:logging.journalMaxBatchSize}”
journalMaxFileSize=”${sys:logging.journalMaxFileSize}”
clientMaxPacketSize=”${sys:logging.clientMaxPacketSize}”
clientConnectTimeoutMs=”${sys:logging.clientConnectTimeout}”
clientSocketTimeoutMs=”${sys:logging.clientSocketTimeout}”
serverAddressPollIntervalMs=”${sys:logging.serverAddressPollInterval}”
serverHeartbeatSendIntervalMs=”${sys:logging.serverHeartbeatSendIntervalMs}”
statisticsPrintIntervalMs=”${sys:logging.statisticsPrintIntervalMs}”>
<PatternLayout pattern=”[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n” />
</Log4J2CloudhubLogAppender>
<?xml version=”1.0" encoding=”UTF-8"?>
<Configuration status=”INFO” name=”cloudhub”
packages=”com.mulesoft.ch.logging.appender,org.apache.logging.log4j”>
<Appenders>
<Log4J2CloudhubLogAppender name=”CLOUDHUB”
addressProvider=”com.mulesoft.ch.logging.DefaultAggregatorAddressProvider”
applicationContext=”com.mulesoft.ch.logging.DefaultApplicationContext”
appendRetryIntervalMs=”${sys:logging.appendRetryInterval}”
appendMaxAttempts=”${sys:logging.appendMaxAttempts}”
batchSendIntervalMs=”${sys:logging.batchSendInterval}”
batchMaxRecords=”${sys:logging.batchMaxRecords}” memBufferMaxSize=”${sys:logging.memBufferMaxSize}”
journalMaxWriteBatchSize=”${sys:logging.journalMaxBatchSize}”
journalMaxFileSize=”${sys:logging.journalMaxFileSize}”
clientMaxPacketSize=”${sys:logging.clientMaxPacketSize}”
clientConnectTimeoutMs=”${sys:logging.clientConnectTimeout}”
clientSocketTimeoutMs=”${sys:logging.clientSocketTimeout}”
serverAddressPollIntervalMs=”${sys:logging.serverAddressPollInterval}”
serverHeartbeatSendIntervalMs=”${sys:logging.serverHeartbeatSendIntervalMs}”
statisticsPrintIntervalMs=”${sys:logging.statisticsPrintIntervalMs}”>
<PatternLayout pattern=”[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n” />
</Log4J2CloudhubLogAppender>
<Http name=”ELK-Cloud” url=”https://bf1718d968f54023b4b4e75badd4fee2.us-east-1.aws.found.io:9243/applogs/_doc">
<Property name=”Authorization” value=”Basic ZWxhc3RpYzoxREd6b0RTMWdTNTdNYlJMbjJ5S2UyU1k=” />
<Property name=”Content-Type” value=”application/json” />
<PatternLayout pattern=’%msg’ />
</Http>
</Appenders>
<Loggers>
<AsyncLogger name=”org.mule.runtime.core.internal.processor.LoggerMessageProcessor” level=”INFO” />
<AsyncLogger name=”com.mulesoft.agent” level=”INFO” />
<AsyncRoot level=”INFO”>
<AppenderRef ref=”CLOUDHUB” />
<AppenderRef ref=”ELK-Cloud” />
</AsyncRoot>
<AsyncLogger name=”com.gigaspaces” level=”ERROR” />
<AsyncLogger name=”com.j_spaces” level=”ERROR” />
<AsyncLogger name=”com.sun.jini” level=”ERROR” />
<AsyncLogger name=”net.jini” level=”ERROR” />
<AsyncLogger name=”org.apache” level=”WARN” />
<AsyncLogger name=”org.apache.cxf” level=”WARN” />
<AsyncLogger name=”org.springframework.beans.factory” level=”WARN” />
<AsyncLogger name=”org.mule” level=”INFO” />
<AsyncLogger name=”com.mulesoft” level=”INFO” />
<AsyncLogger name=”org.jetel” level=”WARN” />
<AsyncLogger name=”Tracking” level=”WARN” />
<AsyncLogger name=”org.mule” level=”INFO” />
<AsyncLogger name=”com.mulesoft” level=”INFO” />
<AsyncLogger name=”org.mule.extensions.jms” level=”INFO” />
<AsyncLogger name=”org.mule.service.http.impl.service.HttpMessageLogger” level=”INFO” />
<AsyncLogger name=”org.mule.extension.salesforce” level=”INFO” />
<AsyncLogger name=”org.mule.extension.ftp” level=”INFO” />
<AsyncLogger name=”org.mule.extension.sftp” level=”INFO” />
<AsyncLogger name=”com.mulesoft.extension.ftps” level=”INFO” />
<AsyncLogger name=”org.mule.modules.sap” level=”INFO” />
<AsyncLogger name=”com.mulesoft.extension.mq” level=”INFO” />
<AsyncLogger name=”com.mulesoft.mq” level=”INFO” />
<AsyncLogger name=”org.mule.extension.db” level=”INFO” />
<AsyncLogger name=”httpclient.wire” level=”DEBUG” />
<AsyncLogger name=”org.mule.transport.email” level=”DEBUG” />
</Loggers>
</Configuration>

--

--

--

I am seasoned Integration Architect with around 16+ yrs of exp. I have extensively worked on TIBCO and Mulesoft. Mainly in EAI, ESB, SOA, API and BPM projects.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Searching Algorithms in Python

Firebase Dynamic Links- Everything You Need to Know

L&D series: Generate GitHub SSH key

How C++ Part 1: How Your Cpp Code Turn Into Program ?

Getting Started with AWS Cognito - Step by Step Guide for Beginners

Web Scraping Yelp with Beautiful Soup

How 1 Change Decreased Clone Times on CI/CD Pipelines by 90%

Python subprocess module

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Mazhar Ansari

Mazhar Ansari

I am seasoned Integration Architect with around 16+ yrs of exp. I have extensively worked on TIBCO and Mulesoft. Mainly in EAI, ESB, SOA, API and BPM projects.

More from Medium

Deploying an API to MuleSoft Flex Gateway running in Oracle Container Engine for Kubernetes

S3 Access Points

Scaling your applications with Auto Scaling on AWS

How to update Appsettings.json based on deployment environment using yaml.