Updating SOA Polling Frequency through WLST

Have you ever wondered how you can update the PollingFrequency of an inbound adapter such as the FileAdapter or FtpAdapter using WLST?

Taking a look at an Oracle SOA 12c composite in the EM console, when you click on your service, you can navigate to the Properties tab to view and update the polling frequency of the inbound adapter at any time during runtime.

We experienced a strange issue. For example, the inbound FileAdapter would be configured to poll the folder every 10 seconds. However, we noticed that in some cases (which we can't explain), the file never gets picked up. It may sit there for hours. A restart of the SOA managed servers took care of this. We also found out that updating the polling frequency re-triggers or re-initializes the inbound file polling for some odd reason.

So the scripts below check for files that have been sitting for a few minutes (that should have been polled), and updates the polling frequency automatically in hopes that the inbound FileAdapter is re-triggered or re-initialized.

Export Composite List Once Per Day

This section describes how the list of composites is exported from the SOA server to a flat file once a day.

1. Cron Job

In cron, I schedule a job to run once a day at 12:05am to export the list of composites. This cron job calls a shell script.

0 5 * * * /home/oracle/scripts/ExportPollingFrequencyComposites.sh prod >> /home/oracle/scripts/ExportPollingFrequencyComposites.log 2>&1

2. ExportPollingFrequencyComposites.log

This cron entry dumps the output of the shell script in ExportPollingFrequencyComposites.log, but this log is mostly used to see if you have errors during execution. An example of this output is:

--------------------------------------------------------------------------------
START: Wed Jul 28 03:00:01 EDT 2021

----------------------------------------
 Export list of composites (/tmp/ahmed_wlst.log)
----------------------------------------

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Connecting to t3://soaprod:8001 with userid soaadmin ...
Successfully connected to managed Server "soa_server1" that belongs to domain "soa_domain".

Warning: An insecure protocol was used to connect to the server.
To ensure on-the-wire security, the SSL port or Admin port should be used instead.

Location changed to custom tree. This is a writable tree with No root.
For more help, use help('custom')

drw-   oracle.soa.config:SCAComposite.SCAService=Get_File,revision=1.0,name=AdapterBinding,partition=Default,SCAComposite="HelloWorld",label=soa_5f753a29-3bf9-40e6-97b0-7949c68dafde,j2eeType=SCAComposite.SCAService.SCABinding,Application=soa-infra
.
.
.

WLST output will no longer be redirected to /tmp/ahmed_wlst.log.
Disconnected from weblogic server: soa_server1


Exiting WebLogic Scripting Tool.


----------------------------------------
Create file with only relevant composites (ExportPollingFrequencyComposites.lst)
----------------------------------------

----------------------------------------
 Show contents of ExportPollingFrequencyComposites.lst
----------------------------------------
END: Wed Jul 28 03:00:19 EDT 2021

3. ExportPollingFrequencyComposites.sh

The shell script ExportPollingFrequencyComposites.sh connects to the SOA server and executes a WLST script ExportPollingFrequencyComposites.py to dump the entire list of composites in /tmp/ahmed_wlst.log.

The arrays in this script (under the section 'COMPOSITE FILTER LIST') removes all exported entries except those explicitly listed in the array. So if there are 100 composites deployed to the SOA server, the final list will only have those that are listed here. The final file is ExportPollingFrequencyComposites.lst.

#!/bin/bash

################################################################################
#
# FILENAME:     ExportPollingFrequencyComposites.sh
# DESCRIPTION:  Exports SOA composite list for the composites below from WLST, to be used in Check script
# AUTHOR:       Ahmed Aboulnaga
# LAST UPDATED: 2020-06-27
# DETAILS:      Called by cron every 24 hours (since it doesn't change often)
#
################################################################################

#----------------------------------------
# SOA 12c ENVIRONMENT SETTINGS
#----------------------------------------
export ORACLE_BASE=/u01/app/oracle
export ORAINVENTORY=/home/oracle/oraInventory
export ORACLE_HOSTNAME=`hostname`
export ORACLE_TERM=xterm
export JAVA_HOME=$ORACLE_BASE/latest
export MW_HOME=$ORACLE_BASE/middleware
export WL_HOME=$MW_HOME/wlserver
export ORACLE_HOME=/u01/app/oracle/middleware
export DOMAIN=soa_domain
export DOMAIN_HOME=/u01/app/oracle/domains/${DOMAIN}
export PATH=$JAVA_HOME/bin:$ORACLE_HOME/OPatch:$PATH:/home/oracle/admin:.

# ----------------------------------------
# SET VARIABLES
# ----------------------------------------
scriptsPath=/home/oracle/scripts

# ----------------------------------------
# HELP
# ----------------------------------------
if [ "$arg" == "--help" ] || [ "$arg" == "-h" ] || [ ${#} != 1 ]; then
  echo ""
  echo "Usage: ./ExportPollingFrequencyComposites.sh "
  echo ""
  echo "Examples:"
  echo "  ./ExportPollingFrequencyComposites.sh test"
  echo ""
  exit
fi

# ----------------------------------------
# INPUT
# ----------------------------------------
varEnv=$1
case "$varEnv" in
  "test")
    varHostname=soatest
    varUsername=soaadmin
    varPassword=welcome1
    ;;
  "prod")
    varHostname=soaprod
    varUsername=soaadmin
    varPassword=welcome1
    ;;
  *)
    echo "Unrecognized environment"
    exit
    ;;
esac

# ----------------------------------------
# COMPOSITE FILTER LIST
# ----------------------------------------

arrayComposites=(
        HelloWorld
        HelloWorld2
        )

arrayRevisions=(
        1.0
        1.0
        )

arrayServices=(
        Get_File
        Get_File2
        )

arrayPollingFrequency=(
        18
        10
        )

arrayFolder=(
        /u01/app/oracle/share/inbound/HelloWorld
        /u01/app/oracle/share/inbound/HelloWorld2
        )

arrayFile=(
        *.csv
        *.csv
        )

echo "--------------------------------------------------------------------------------"
echo "START: `date`"

echo ""
echo "----------------------------------------"
echo " Export list of composites (/tmp/ahmed_wlst.log)"
echo "----------------------------------------"
$MW_HOME/oracle_common/common/bin/wlst.sh ${scriptsPath}/ExportPollingFrequencyComposites.py $varUsername $varPassword $varHostname

echo ""
echo "----------------------------------------"
echo " Create file with only relevant composites (ExportPollingFrequencyComposites.lst)"
echo "----------------------------------------"
rm -f ${scriptsPath}/ExportPollingFrequencyComposites.lst
for i in ${!arrayComposites[@]}; do
  cat /tmp/ahmed_wlst.log | grep "${arrayComposites[$i]}" | grep "revision=${arrayRevisions[$i]}" | grep "${arrayServices[$i]}" | grep "oracle.soa.config:SCAComposite.SCAService" | awk '{print $2}' >> ${scriptsPath}/ExportPollingFrequencyComposites.lst
done

echo ""
echo "----------------------------------------"
echo " Show contents of ExportPollingFrequencyComposites.lst"
echo "----------------------------------------"

echo "END: `date`"

4. ExportPollingFrequencyComposites.py

This is the WLST script that lists all composites and dumps them into /tmp/ahmed_wlst.log:

weblogicUsername = sys.argv[1]
weblogicPassword = sys.argv[2]
weblogicHost = sys.argv[3]

connect(weblogicUsername,weblogicPassword,'t3://' + weblogicHost + ':8001')

custom()
cd('oracle.soa.config')
redirect('/tmp/ahmed_wlst.log')
ls()
stopRedirect()

disconnect()
exit()

The composites listed here are in a very particular and specific format which we'll soon need.

5. ExportPollingFrequencyComposites.lst

The final output of this daily dump is maintained in ExportPollingFrequencyComposites.lst, an example which is shown here. This is the list of composites we want to focus on (based on the array in the .sh script).

oracle.soa.config:SCAComposite.SCAService=Get_File,revision=1.0,name=AdapterBinding,partition=Default,SCAComposite="HelloWorld",label=soa_4c33e070-eb79-443e-bcc4-1b2651bd880d,j2eeType=SCAComposite.SCAService.SCABinding,Application=soa-infra
oracle.soa.config:SCAComposite.SCAService=Get_File2,revision=1.0,name=AdapterBinding,partition=Default,SCAComposite="HelloWorld2",label=soa_9384e070-eb79-935s-xyz4-1b2651b9987d,j2eeType=SCAComposite.SCAService.SCABinding,Application=soa-infra

Check Files Every 5 Minutes and (Maybe) Update Polling Frequency

Now these next scripts go through the list of composites, checks if there are files in the directory older than 5 minutes, and if so updates the polling frequency.

1. Cron Job

In cron, I schedule a job to run every 5 minutes to check if there are files older than 5 minutes, and if so will update the polling frequency of the inbound FileAdapter. This cron job calls a shell script.

*/5 * * * * /home/oracle/scripts/CheckPollingFrequency.sh prod >> /home/oracle/scripts/CheckPollingFrequency.log 2>&1

2. CheckPollingFrequency.log

This cron entry dumps the output of the shell script in CheckPollingFrequency.log, but this log is mostly used to see if you have errors during execution. An example of this output is:

--------------------------------------------------------------------------------
[INFO] Start process...         Wed Jul 28 16:00:01 EDT 2021
[INFO] Checking composite #1... HelloWorld | Get_File | 1.0
[INFO] Checking these files...  /u01/app/oracle/share/inbound/HelloWorld/*.csv
[INFO] Checking composite #2... HelloWorld2 | Get_File2 | 1.0
[INFO] Checking these files...  /u01/app/oracle/share/inbound/HelloWorld2/*.csv
|
| ******************************
| *           WARNING          *
| * FILES OLDER THAN 5 MINUTES *
| ******************************
|
| COMPOSITE: HelloWorld2
| SERVICE:   Get_File2
| REVISION:  1.0
|
| FILES:
| Jul 21 11:11 /u01/app/oracle/share/inbound/HelloWorld2/data.csv
|
[CRITICAL] Setting polling frequency to 10 for composite... HelloWorld2 | Get_File2 | 1.0

[INFO] End process...           Wed Jul 28 16:00:02 EDT 2021

3. CheckPollingFrequency.sh

The shell script CheckPollingFrequency.sh loops through all folders and checks if any file is older than 5 minutes. If so, it calls the WLST CheckPollingFrequency.py to update the polling frequency of the respective composite.

#!/bin/bash

################################################################################
#
# FILENAME:     CheckPollingFrequency.sh
# DESCRIPTION:  Updates SOA composite PollingFrequency if files for FileAdapter not picked up in 5 minutes
# AUTHOR:       Ahmed Aboulnaga
# LAST UPDATED: 2020-12-16
# DETAILS:      - Called by cron every 5 minutes
#               - Requires the file 'ExportPollingFrequencyComposites.lst' to exist
#               - File 'ExportPollingFrequencyComposites.lst' created by 'ExportPollingFrequencyComposites.sh'
#
################################################################################

#----------------------------------------
# SOA 12c ENVIRONMENT SETTINGS
#----------------------------------------
export ORACLE_BASE=/u01/app/oracle
export ORAINVENTORY=/export/home/oracle/oraInventory
export ORACLE_HOSTNAME=`hostname`
export ORACLE_TERM=xterm
export JAVA_HOME=$ORACLE_BASE/latest
export MW_HOME=$ORACLE_BASE/middleware
export WL_HOME=$MW_HOME/wlserver
export ORACLE_HOME=/u01/app/oracle/middleware
export DOMAIN=soa_domain
export DOMAIN_HOME=/u01/app/oracle/domains/${DOMAIN}
export PATH=$JAVA_HOME/bin:$ORACLE_HOME/OPatch:$PATH:/home/oracle/admin:.

#----------------------------------------
# SET VARIABLES
#----------------------------------------
varOlderThanMinutes=5
scriptsPath=/home/oracle/scripts
EMAILS=ahmed@revelationtech.com

# ----------------------------------------
# INPUT
# ----------------------------------------
if [ "$arg" == "--help" ] || [ "$arg" == "-h" ] || [ ${#} != 1 ]; then
  echo ""
  echo "Usage: ./CheckPollingFrequencyComposites.sh "
  echo ""
  echo "Examples:"
  echo "  ./CheckPollingFrequencyComposites.sh test"
  echo ""
  exit
fi

# ----------------------------------------
# CHECK FILE EXISTENCE
# ----------------------------------------
if [ ! -f "${scriptsPath}/ExportPollingFrequencyComposites.lst" ]; then
  echo "[INFO] Start process...         `date`"
  echo "[ERROR] File 'ExportPollingFrequencyComposites.lst' does not exist..."
  echo "[INFO] End process...           `date`"
  exit
fi

# ----------------------------------------
# ENVIRONMENT SETTINGS
# ----------------------------------------
varEnv=$1
case "$varEnv" in
  "test")
    varHostname=soatest
    varUsername=soaadmin
    varPassword=welcome1
    ;;
  "prod")
    varHostname=soaprod
    varUsername=soaadmin
    varPassword=welcome1
    ;;
  *)
    echo "Unrecognized environment"
    exit
    ;;
esac

# ----------------------------------------
# SOA COMPOSITES WITH INBOUND FILEADAPTER TO CHECK
# ----------------------------------------

arrayComposites=(
        HelloWorld
        HelloWorld2
        )

arrayRevisions=(
        1.0
        1.0
        )

arrayServices=(
        Get_File
        Get_File2
        )

arrayPollingFrequency=(
        18
        10
        )

arrayFolder=(
        /u01/app/oracle/share/inbound/HelloWorld
        /u01/app/oracle/share/inbound/HelloWorld2
        )

arrayFile=(
        *.csv
        *.csv
        )

#----------------------------------------
# Loop through each composite in ExportPollingFrequencyComposites.lst
#----------------------------------------
varLoopCount=0
echo "--------------------------------------------------------------------------------"
echo "[INFO] Start process...         `date`"

while read f; do

  varLoopCount=$((varLoopCount+1))
  varCurrentLine=${f}

  #----------------------------------------
  # Extract Service and Composite and Composite Revision from current line
  #----------------------------------------
  varCurrentLineService=`echo $varCurrentLine | cut -f1 -d, | cut -f2 -d=`
  varCurrentLineComposite=`echo $varCurrentLine | cut -f5 -d, | cut -f2 -d= | cut -d "\"" -f 2`
  varCurrentLineRevision=`echo $varCurrentLine | cut -f2 -d, | cut -f2 -d=`

  #echo "------ ${varCurrentLine}"
  echo "[INFO] Checking composite #${varLoopCount}... ${varCurrentLineComposite} | ${varCurrentLineService} | ${varCurrentLineRevision}"

  #----------------------------------------
  # Loop through array
  #----------------------------------------
  for i in ${!arrayComposites[@]}; do

    #----------------------------------------
    # If current line matches array values
    #----------------------------------------
    if [ "${varCurrentLineService}" == "${arrayServices[$i]}" ] && [ "${varCurrentLineComposite}" == "${arrayComposites[$i]}" ] && [ "${varCurrentLineRevision}" == "${arrayRevisions[$i]}" ]; then

      #----------------------------------------
      # Extract polling frequency
      #----------------------------------------
      varPollingFrequency=${arrayPollingFrequency[$i]}

      if [ "${arrayFile[$i]}" == "XYZ" ]; then
        echo "[INFO] Checking these files...  ${arrayFolder[$i]}/*.*"
      else
        echo "[INFO] Checking these files...  ${arrayFolder[$i]}/${arrayFile[$i]}"
      fi
      #----------------------------------------
      # If Folder exists
      #----------------------------------------
      if [ -d "${arrayFolder[$i]}" ]; then

        #----------------------------------------
        # If Folder has File types older than 5 minutes
        #----------------------------------------
        if [ "${arrayFile[$i]}" == "XYZ" ]; then
          varStaleFileCount=`find ${arrayFolder[$i]} -maxdepth 1 -mmin +${varOlderThanMinutes} -type f -name "*.*" | wc -l`
        else
          varStaleFileCount=`find ${arrayFolder[$i]} -maxdepth 1 -mmin +${varOlderThanMinutes} -type f -name "${arrayFile[$i]}" | wc -l`
        fi

        if [ ${varStaleFileCount} -gt 0 ] ; then

          #----------------------------------------
          # Log a warning
          #----------------------------------------
          echo "| "
          echo "| ******************************"
          echo "| *           WARNING          *"
          echo "| * FILES OLDER THAN 5 MINUTES *"
          echo "| ******************************"
          echo "| "
          echo "| COMPOSITE: ${varCurrentLineComposite}"
          echo "| SERVICE:   ${varCurrentLineService}"
          echo "| REVISION:  ${varCurrentLineRevision}"
          echo "| "
          echo "| FILES:"
          if [ "${arrayFile[$i]}" == "XYZ" ]; then
            find ${arrayFolder[$i]} -maxdepth 1 -mmin +${varOlderThanMinutes} -type f -name "*.*" -ls | awk '{print "| " $8 " " $9 " " $10 " " $11}'
          else
            find ${arrayFolder[$i]} -maxdepth 1 -mmin +${varOlderThanMinutes} -type f -name "${arrayFile[$i]}" -ls | awk '{print "| " $8 " " $9 " " $10 " " $11}'
          fi
          echo "|"

          #----------------------------------------
          # Touch the files
          #----------------------------------------
          if [ "${arrayFile[$i]}" == "XYZ" ]; then
            for ix in `find ${arrayFolder[$i]} -maxdepth 1 -mmin +${varOlderThanMinutes} -type f -name "*.*" -ls | awk '{print $11}'`
            do
              touch ${ix}
            done
          else
            for ix in `find ${arrayFolder[$i]} -maxdepth 1 -mmin +${varOlderThanMinutes} -type f -name "${arrayFile[$i]}" -ls | awk '{print $11}'`
            do
              touch ${ix}
            done
          fi

          #----------------------------------------
          # Call WLST script to update polling frequency
          #----------------------------------------
          echo "[CRITICAL] Setting polling frequency to ${varPollingFrequency} for composite... ${varCurrentLineComposite} | ${varCurrentLineService} | ${varCurrentLineRevision}"

          $MW_HOME/oracle_common/common/bin/wlst.sh ${scriptsPath}/CheckPollingFrequency.py $varUsername $varPassword $varHostname $varCurrentLine $varPollingFrequency

          #----------------------------------------
          # Send custom email
          #----------------------------------------
          if [ "${varCurrentLineComposite}" == "AllvacLims_BPEL" ] || [ "${varCurrentLineComposite}" == "FileProcessorUtilityUsingShell" ]; then
            mail -s "EM Event: Critical:${HOSTNAME} - PollingFrequency updated (${varCurrentLineComposite}|${varCurrentLineService}|${varCurrentLineRevision})" ${EMAILS_MORE}
          else
            mail -s "EM Event: Critical:${HOSTNAME} - PollingFrequency updated (${varCurrentLineComposite}|${varCurrentLineService}|${varCurrentLineRevision})" ${EMAILS}
          fi

        fi

      else
        echo "[WARNING] Skipping, folder does not exist... ${arrayFolder[$i]}"

      fi
    fi

  done

done < ${scriptsPath}/ExportPollingFrequencyComposites.lst
echo "[INFO] End process...           `date`"

4. CheckPollingFrequency.py

This WLST script updates the polling frequency of the composite. The 5 input parameters are passed by the .sh script.

weblogicUsername = sys.argv[1]
weblogicPassword = sys.argv[2]
weblogicHost = sys.argv[3]
weblogicComposite = sys.argv[4]
weblogicPollingFrequency = sys.argv[5]

import re
import tempfile

connect(weblogicUsername,weblogicPassword,'t3://' + weblogicHost + ':8001')

custom()
cd('oracle.soa.config')

myComposite = ObjectName(weblogicComposite)

# DEBUG
# myComposite = ObjectName('oracle.soa.config:SCAComposite.SCAService=Get_File,revision=1.0,name=AdapterBinding,partition=Default,SCAComposite="HelloWorld",label=soa_6d8fbdcb-d6be-49ed-8a73-ea1dc159427f,j2eeType=SCAComposite.SCAService.SCABinding,Application=soa-infra')

print '-----------myComposite------------------'
print myComposite

print '----------Before Properties-------------'
print mbs.getAttribute(myComposite, 'Properties')

print '-----------Set Properties---------------'
#params = ['PollingFrequency','55']
params = ['PollingFrequency',weblogicPollingFrequency]
sign = ['java.lang.String','java.lang.String']
mbs.invoke(myComposite, 'setStringProperty', params, sign)

print '-----------Save Properties--------------'
mbs.invoke(myComposite, 'save', None, None)

print '-----------After Changes-----------------'
print mbs.getAttribute(myComposite, 'Properties')

disconnect()
exit()

This is not a particularly easy set of scripts to understand at first glance, so feel free to reach out to me for further explanation.