August 12, 2019

Bulk Recovery of Fault Error Instances

By: Viswanatha Basavalingappa | Consulting Member of Technical Staff


One of the most common requirements of enterprise integration use case is error management. it is very critical for customers to manage the recoverable errors seem-less and in automated fashion.

What are Recoverable Fault Errors?

All faulted instances in asynchronous flows in Oracle Integration Cloud Service are recoverable and can be resubmitted. Synchronous flows cannot be resubmitted. You can resubmit errors in the following ways:

  • Single failed message resubmissions

  • Bulk failed message resubmissions

Today user can manually resubmit failed messages one by one from console. here are the steps do so,  

In this blog, what we are focusing is how user can create integration flow that does Bulk resubmit those type of Recoverable fault error instance in automated fashion using Oracle Integration flow it self. 

Here are the Steps to Create Integration flow to implement the Auto Bulk Recovery Errors 

STEP1: Create New Scheduled Orchestration Flow 

STEP2: Add Schedule Parameters 

  • strErrorQueryFilter : Filter parameter.
    • Valid values:  timewindow: 1h, 6h, 1d, 2d, 3d, RETENTIONPERIOD. Default is 1h.
    • code: integration code version: integration version
    • id: error id(instance id)
    • primaryValue: value of primary tracking variable
    • secondaryValue: value of secondary tracking variable
  • strMaxBatchSize: Max Number of Error Instance can resubmit per batch invoke
  • strMinBatchSize: (10): Minimum Number of Error Instance to be found for batch resubmit
  • strRetryCount:  (3) Max number of Times retry attempt can be done for any given Error instance
  • strMaxThershold: (500) Max number of errors found 

STEP3: Update the Query Filter to Include only Recovery Errors


STEP4: Query All Recoverable Error Instances in the System for a given Query Filter from above parameter.

GET /ic/api/integration/v1/monitoring/errors?q=strErrorQueryFilter

STEP4a> : if Total Recovery Error Instances Found totalResults > strMaxThershold defined in schedule parameter, then Send a Notification to Stack hold to take action .i,e verify is there a known system down or other target systems are affected if so if they fixed .

STEP4b> : if Total Recovery Error Instances Found totalResults <= 0 then End Flow

STEP4c> else: Continue to resubmit the Found Errors in a batch

NOTE: Since we have to limit the number of errors can be re-submit in one batch to certain amount since suggested value of 50. 

process the found errors in batch of 50 in a while loop as below 

STEP5: Define Loop index Variable.

  • strStartIndex = 0
  • strTotalCount = totalResults

STEP6: Start While Loop 

Where  'number( $strStartIndex) <= number( $strTotalCount)'

STEP6a: Query Recovery Errors of Max Limit == strMaxBatchSize ( 50)

STEP6a1 : totalResults <  strMinBatchSize : skip the batch re-submit

STEP6a2 : else: Invoke Bulk Re-submit Error API.

    Map list of Fault IDs to Bulk Submit: where Filter out the  retryCount < = $strMaxRetryAttempt

  POST - /ic/api/integration/v1/monitoring/errors/resubmit


STEP6b: Update Start index Variable. - string(number($strStartIndex) + number($strBatchSize))

STEP6c: Update the Bulk Submit Recovery Job ID to List.

Continue the While Loop.

STEP8 :  at the end of Integration one can module the Integration to 

Option1: Send Email Notification with All Recovery Job IDs for tracking, end user can monitor the recovery Jobs in OIC Console UI.

Option2: Invoke Child Job where user can integrate the process to take additional actions