8. Error Handling¶

Revision:	1.2
Status:	Released
Repository:	https://gitlab.eso.org/cii/srv/cii-srv
Project:	ELT CII
Folder:	/trunk/deliverables/phase7
Document ID:	CSL-MAN-19-172698
File:	MAN-ELT-ErrorHandling
Owner:	Borut Terpinc
Last modification:	November 23, 2020
Created:	April 2, 2019

Prepared by

Reviewed by

Approved by

Borut Terpinc (CSL SWE)

Jan Pribošek (CSL SWE)

Miha Vitorovič (CSL)

Gregor Čuk (CSL)

Document History

Revision	Date	Changed/rev iewed	Section(s)	Modificatio n
0.9	9.4.2019	bterpinc	All	Created.
1.0	16.9.2019	bterpinc	All	Updated after internal review.
1.1	10.2.2020	bterpinc	All	Major update of all sections after Topical review comments.
1.2	20.11.2020	bterpinc	2.2	Added explanation about the error severity.

Confidentiality

This document is classified as a confidential document. As such, it or parts thereof must not be made accessible to anyone not listed in the Audience section, neither in electronic nor in any other form.

Scope

This document is an error handling manual for the ELT Core Integration Infrastructure Software project.

Audience

This document is aimed at those Cosylab and ESO employees involved with the ELT Core Integration Infrastructure Software project, as well as other Users and Maintainers of the ELT Core Integration Infrastructure Software.

Glossary of Terms

API	Application Programmers Interface
CII	Core Integration Infrastructure
GUI	Graphical User Interface
ICD	Interface Control Document
IDE	Integrated development environment
MAL	Middleware Abstraction Layer
SVN	Subversion version control system

References

Cosylab, Interface Control Document, Specification, CSL-DOC-17-147262, version 1.3
Cosylab, Error handling design document, CSL-DOC-17-167386, version 1.2
ELT Config, Error Handling Transfer document, CSL-DOC-19-172644, version 1.5
ElasticSearch Query DSL, https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html
ElasticSearch, Standard Analyzer, https://www.elastic.co/guide/en/elasticsearch/reference/6.8/analysis-standard-analyzer.html

8.1. Overview¶

This document is a user manual for CII Error handling system.

8.2. Introduction¶

Error Handling in CII provides the APIs and tools necessary to conveniently implement a common error handling mechanism across all EELT Control System applications and thus streamline the diagnosis of abnormal behavior of the system.

Figure 2‑1 Interaction diagram of error handling

In general, error handling covers the following error management activities:

Error detection: language-specific try/catch mechanisms provide error detection.

Error handling: errors are based on exceptions. The name of the exception with the namespace defines the error type and error severity. Catching errors is based on the try/catch mechanism of each language. Error grouping is supported by exception inheritance. Error message and description is a part of exception class.

Propagation of error information: Error information is contained inside the exception entities. This information is passed between the different program parts. Passing information over the network is realized with ICD-based exceptions.

Administration and collection of all-important information: Definition of the error information is contained inside the exception class. New classes can be added and changed and it is also possible to change existing exception classes. All error information can be logged to the logging system.

Error messages will be displayed to the user: A QT GUI error display widget provides a GUI component for displaying the error messages. In the case of an error, the error message is displayed in a pop-up dialog.

All components of the error handling are provided for 3 languages: Java, CPP and Python.

8.2.1. Standard workflow¶

Figure 2‑2 Error code repository and workflow

Figure 2‑2 presents the typical workflow for storing the exception information in a central repository. It is advised to have one or more general maintainers (lead developers) of the error code system. The maintainer is responsible for the error code system integrity and has an overview of all the exceptions in the system (or a specific section).

Whenever the developers need a specific exception, they first checkout the latest version of the SVN module where needed exception is located. They then use grep, find or any other file text search tool to find the desired exception. They can also browse the folder tree to find the exceptions that belong to specific groups. A helper search script searchExceptions.sh is provided to aid the developer when searching. If a developer cannot find the needed exception, one creates a new exception that is stored in a separate development branch. The new exception can be used immediately in the development branch. Then the developer commits the proposed new exception to central storage in the separate branch. The developer informs the maintainer about the new exception. The maintainer reviews the new exception and either merges the exception into the trunk or proposes the use of an existing exception from the trunk to the developer. When a new exception is merged into the system, the maintainer sends the message to the developer. The developer checks out the latest version of the trunk and uses the new exception from the trunk.

If a change to the exception information is needed, the same process as for creating an exception is used; the only difference is that the existing exception is modified and stored to the development branch. Subsequent steps are identical.

8.2.2. CII Exceptions¶

The CII error handling mechanism is based on CiiException, which is the base exception from which all other exceptions are derived. The CiiException derives from the special ICD defined CiiSerializedException, which provides serialization (2.4). Derived from the CiiException, two additional exceptions are defined as the part of the error handling system: CiiError and CiiBaseException. These two exceptions correspond to the error severity:

Normal - CiiBaseException – these exceptions cover the errors which can be recovered, and can be handled at higher software layers.
Error - CiiError – these exceptions cover the unrecoverable errors. When unrecoverable error happens, the program shall exist with the corresponding error message.

The severity of the error is defined by the exception name and the parent exception from which all errors of the selected severity are defined. The exception and error severities are pre-defined with exception names. All exceptions defined for severity »error« must contain the word error in the exception class name. All exceptions with »normal« severity must contain the word exception in the class name. Exceptions that define error severity are based on CiiError exception and exceptions with normal severity must derive from CiiBaseException.

All user defined exceptions must derive from the CiiBaseException or from CiiError.

Figure 2‑3 CII base exceptions and errors

Figure 2‑3 shows the class inheritance diagram of the base errors and exceptions. The exceptions defined by the particular CII service all derive from the CiiBaseException for »normal« severity and from CiiError for »error« severity. Every CII service also defines root exceptions for the two severities (e.q., CiiOLDBException, CiiOLDBError). From these two exceptions, all the other exceptions for a particular system are derived.

8.2.2.1. Error stack¶

Single exception instance can nest exceptions and create an error stack. The stack is created by construction exceptions with the included information about the nested exception. A hierarchy of nested exceptions constructs the exception stack. Every nested exception presents an entry to the exception error stack. Exceptions (entries to the error stack) can only be added to the error stack, they cannot be changed or deleted. The error stack is automatically removed when reference to the exception with all the nested exceptions is lost.

In C++, a special throw mechanism must be used to create a stack of errors.

Whenever there is a need for the stack to be logged, an explicit entry to the log system must be programmed by the developer.

The examples of creating the exception stack can be found in 4.6.1.

8.2.2.2. Indexing Service¶

The Indexing service is a Java-based program that collects information about the exceptions and stores them in the Elasticsearch. The Indexing Service will traverse the SVN directory where exceptions are stored and crawl over the source code to extract the following information from the exceptions:

error type,

error message,

error description,

author.

Extraction uses the flowing rules:

The error type is extracted from the exception name and folder path.

The error description is extracted from the class comment. The description starts with the @desc tag.

The author of the exception is extracted from the class comment. It starts with the @author tag.

The error message is extracted from the exception class member variable named message. The variable is of type string.

The indexing service expects all information other than the description to be stored in one line.

To provide namespace information for the error type, the directory structure information is used. As the exceptions must be properly structured in the named folders that reflect the namespace, this information will provide the proper information for the error type namespace.

8.2.3. Conditions API¶

Error handling also provides conditions API. The purpose of the API is to provide a convenience API to enable checking of pre and post conditions, and method invariants. The API also supports enabling and disabling conditions and fail-fast checks. API provides static methods for verifying arguments, null values, states and valid index positions in lists and arrays. The methods, in general, accept one or more arguments for checking the condition and an argument that defines the exception type. The API interacts with the error handling system.

Whenever the condition is not met, the API will throw an exception. API contains an option for setting the fail-fast mode. When the fail-fast mode is enabled, the language-specific root exception will be thrown. This will cause the application to exit with the defined error message.

The error data is centrally stored in SVN. Chapter A.3 contains a description of Conditions API methods.

8.2.4. Serialization¶

Transportation of the exceptions over the distributed processes, written in different programming languages is supported with exception serialization. Serialized exceptions are transferred over MAL, which provides serialization of exceptions into plain objects. Therefore, exceptions are passed over the network with information serialized to string.

MAL uses ICD schema to define the exception structure. Custom user-defined ICD exceptions with desired information can be created. For these exceptions, there is no mechanism to implicitly transfer the data from the CiiException based exceptions to the ICD exception. Exception data can be read from the CiiException based exceptions and used to fill the ICD user-defined member fields. When an ICD based exception is transferred over the network, it is the developer’s responsibility to catch the exception and manage the data transfer and pass the exception to the higher levels of error handling.

For simplicity of use, the ICD CiiSerializedException is defined. This is a special ICD exception with hard-coded additions. This exception has predefined member fields, which carry the data of the exception. CiiSerializedException is also a base exception for all CiiExceptions. This special exception simplifies transferring exceptions over the network from the sender side, as no data copying between exceptions is needed. All CiiExceptions are derived from the CiiSerializedException, this means that CiiSerializedException will automatically catch all user-defined CiiExceptions. All exceptions that use CiiException will be automatically transferred over the network as CiiSerializedException.

8.3. Installation¶

Error Handling provides the following components that need to be installed:

Error indexer
Error handling dialog widget
Conditions API

See Configuration and Error Handling Transfer Document [3] for instructions to install the error handling system.

8.3.1. Prerequisites and waf modules¶

Error Handling uses SVN as central storage for exceptions. Error Handling also uses Elasticsearch as an engine for more structured searching of the exceptions. The two products must be installed and accessible from the local hosts to use the Error Handling:

SVN server,
Elasticsearch.

Error handling uses the following CII modules:

elt-mal: MAL, MAL ZPB.
client-api: Conditions API, Error Handling API.
srv-error-indexer: Error indexer, search script.
elt-qt-widgets: Error handling dialog widget.

8.4. Usage¶

The example in this manual is based on CLI SVN interface. Standard language IDE’s (Eclipse, IntelliJ Idea, QT Creator) can be used for writing exception definitions and to communicate with the SVN server. Examples of IDE use are not part of this manual.

8.4.1. Includes/Imports¶

For basic usage of the Error Handling, the following programming language imports must be added:

8.4.1.1. Java¶

import elt.error.CiiError;
import elt.error.CiiBaseException;
import elt.error.CiiConditions;

8.4.1.2. CPP¶

#include <CiiException.hpp>
#include <CiiConditions.hpp>

8.4.1.3. Python¶

import elt.error

8.4.1.4. QT¶

#include <CiiException.hpp>
#include <CiiExceptionDialog.hpp>

8.4.2. Creating an SVN repository¶

To use the error codes system an SVN repository must be created, which will act as a storage for all CII Exceptions. It is expected that this repository is already created by the lead developer or repository maintainer. The developer should consult the responsible person for the information about the location.

In this manual we will create a repository in an SVN server hosted at https://svnServer, at /pathToErrorCodes/errorCodes path. The repository can be created with the following command:

svn mkdir -m "Create error codes directory" https://svnServer/pathToErrorCodes/errorCodes --username myusername

In the errorCodes repository, the basic error codes system structure must be created. Folders can be created in the same manner as the root folder, or a local copy of structure can be prepared and committed to the server. The following structure must be created:

errorCodes/trunk/java

errorCodes/trunk/cpp

errorCodes/trunk/python

errorCodes/branches

Note: The root folder in SVN for the exceptions can be changed to the specific path (e.g. p8/trunk/CONTROL-SYSTEMS/CentralControlSystem/CII/CODE/srv-error-indexer/sampleErrorCodeTestRepo/). However, the structure of folders after the root folder is fixed and cannot be changed in order for Indexing service (6) to properly collect the error codes from SVN repository. Indexing service expects trunk folder and subfolders java, cpp and python. The names of the subfolder’s are a part of indexing service source code. A small change in Indexing service code must be made to use the different names for the folder structure.

8.4.3. Creating the exceptions¶

Exceptions must be created in the folder reflecting the namespace and with a file name reflecting the exception class name. Multiple exceptions with the same name are allowed, if they are put in separate namespaces. Exception name together with the namespace defines the error type and severity.

The workflow of error codes is described in section 2.1. Shortly, a developer must create exceptions in his own branch, which is then checked by the maintainer and merged into the trunk.

Note: It is expected that the development environment was properly set and that the variable $INTROOT points to the root location where all the CII components are installed (command “module load introot eeltdev” was run).

If the SVN errorCodes folder is not present on local developer’s computer, we first need to check-out the errorCodes directory from the SVN.

cd $INTROOT/sources/
svn co https://svnServer/pathToErrorCodes/errorCodes --username myusername

New exceptions, defined by a developer, should be created in their own branch. It is suggested to create this branch with a folder named after developer username. For each language a separate folder is required (java, cpp, python).

cd branches/
mkdir developerName
cd developerName
mkdir java cpp python

Exceptions source code must be structured as shown in the following examples. Exception files must be placed in the folders, whose names reflect the namespace. Each single exception is placed in own file with the filename named the same as the exception class.

In the example below we create CiiFileHandlingException, which is in the elt.error.common package. We create this exception for all 3 languages. Folders for this package must be created:

mkdir -p java/src/elt/error/common
mkdir -p python/src/elt/error/common
mkdir -p cpp/elt/error/src/include

Information from the exceptions is parsed by the indexing service. In order for the indexing service to properly parse the exceptions, the exceptions must be placed in the described folders. Also, the structure of exceptions source code must follow the structure presented in the following chapters (4.3.1, 4.3.2, 4.3.3). Information about the author and description of the exception is parsed from the source code comments. The information about the exception message is parsed from the message string.

Examples of the exceptions SVN repository can be found in the srv-error-indexer module under the sampleErrorCodesRepo directory. The following sections show the programming language specific examples.

8.4.3.1. Java¶

To create a Java exception, create CiiFileHandlingException.java file inside the java/src/elt/error/common folder with the desired exception specifics. The following example shows the Java exception structure:

package elt.error.common

/**
 * @desc This is the description of file handling error.
 * @author Somebody@eso.org
 */

public class CiiFileHandlingException extends CiiBaseException {
  public static final String MESSAGE = "Error while handling file: %s";

  public CiiFileHandlingException(String path){
    super(String.format(MESSAGE, path));
  }
}

8.4.3.2. CPP¶

For CPP exceptions to be browsable, they must follow the Java style for exception filenames. Each exception is created in its own .hpp file, with the file named the same as the exception class name. Exception details (description and author) must be added inline in the header code. CPP exceptions are header only based, no .cpp files shall be created.

To create a CPP exception, create CiiFileHandlingException.hpp file inside the cpp/elt/error/common/src/include folder with the desired exception specifics. The following example shows the CPP exception structure:

/**
 * @desc This is the description of file handling error.
 * @author Somebody@eso.org
 */

namespace elt {
namespace error {
namespace common {

class CiiFileHandlingException : public CiiBaseException {

const std::string getMessage(){
  static const std::string& message = "Error while handling file: %s";
  return message;
  //other option for message definition
  //return std::string("Bad version number %i");
}
public:
  CiiFileHandlingException(const char* fileName):
                   CiiException(getMessage(), fileName){}
};

//other option std::string
//if the argument passed to the CII Exception is string it has to be converted to const char*
/*
public:
  CiiFileHandlingException(const std::string& fileName):
                   CiiException(getMessage(), fileName.c_str()){}
}; */
}
}
}

8.4.3.3. Python¶

For Python exceptions to be browsable, they must follow the Java style for exception filenames. Each exception is created in its own .py file, with the file named the same as the exception class name.

To create a Python exception, create CiiFileHandlingException.py file inside the python/src/elt/error/common folder with the desired exception specifics. The following example shows the Python exception structure:

"""
@desc This is the description of file handling error.
@author Somebody@eso.org
"""
class CiiFileHandlingException(CiiBaseException):
  _message = "Error while handling file: %s"
  def __init__(self, arg1):
    super().__init__(_message % arg1)

After the exceptions are created in a separate branch, the developer commits the exceptions to the SVN server and informs the exceptions maintainer about the existence of new exceptions. The maintainer then merges the exception into the trunk.

It is the maintainer’s responsibility to check that the exception structure (folders and namespace) conforms to the convention.

8.4.4. Wrapping native exceptions¶

As native language exceptions follow their own inheritance tree, they cannot be directly caught by the CiiException. For this reason, all the native exceptions must be caught and rethrown with the details passed to the variant of the CiiException. This way information provided by the native exceptions can be part of the CII Exception.

The following example shows how to pass the native exception information to CII Exceptions in Java:

try {
  testList.get(arrayIndex);
} catch (IndexOutOfBoundsException ex) {
   throw new CiiBadIndexException(arrayIndex, ex);
}

Same mechanism should be used in Python and CPP.

8.4.5. Search script¶

Search script searchExceptions.sh is a simple bash script providing capabilities of automatic SVN update, local search and remote search. The script is a wrapper script around the Linux find and grep commands. The script automatically checks out the latest version of errorCodes from the SVN server and executes a search. It also provides search for indexed exceptions in Elasticsearch. While the functionality of the search in Elasticsearch is practically the same as for other find and grep style searches, it provides more structured output with more detailed information about the exceptions.

The script is intended to be used locally by developers. All the settings are embedded directly in the script.

8.4.5.1. Configuring the script¶

To configure the script, open it in a text editor:

nano $INTROOT/bin/searchExceptions.sh

Change the following settings:

REMOTE_REPO: location of the root of remote SVN repository
SVN_USERNAME: username with SVN permissions
SVN_PASSWORD: password for the SVN user
LOCAL_REPO: location of the root of local SVN repository

Example:

REMOTE_REPO="https://svnhq8.hq.eso.org/p8/trunk/CONTROL-SYSTEMS/CentralControlSystem/CII/CODE/srv-error-indexer/sampleErrorCodesRepo"
SVN_USERNAME="myusername"
SVN_PASSWORD="mypassword"
LOCAL_REPO="myLocalRepoRoot/sampleErrorCodesRepo"

8.4.5.2. Usage of the search script¶

A search without a special flag argument will perform a grep style search:

./searchExceptions.sh "Error while handling file:"

The search with the -f argument will execute a find style search:

./searchExceptions.sh -f "CiiFileHandlingException.java"

Direct Elasticsearch index searches can be executed with the -r argument. The following example shows how to search the exception index using the Elasticsearch query:

searchExceptions.sh -r "file error"

This searches for the exceptions that contain either word file or word error in the following fields of the Elasticsearch index:

Name
Folder
Namespace
Author
Description
Message
Language
Comment

A more relevant match (i.e. all or many search words are present) appear sooner in the returned list of results.

To see all functions supported by the script, the -h option can be used:

searchExceptions.sh -h

8.4.6. Exceptions examples¶

The examples show usage of the CII error handling system for Java, CPP and Python. As the majority of native language exception handling is similar across all three languages, a general example is shown only for Java:

public class ErrorHandlingExample {

  //initialize logger
  private static Logger logger =
                 CiiLogManager.getLogger("ErrorHandlingUsageExample");

  public static void main(String[] args) {

    /* Main method is the on the highest level (first) of exception handling.
     * All unhandled exception will end the application.
     */

    //initialize example object
    ErrorHandlingExample ex = new ErrorHandlingExample();

    /* Handle the exception from the runProductionArrayElementCode() and
     * log the current exception to log (Level.WARNING).
     */
    try {
      ex.runProductionArrayElementCode();
    } catch (CiiException e) {
      /* Exception is treated as warning. Current exception stack
       * is logged on a warning.
       * Exception is not passed to higher level, so the stack is cleared.
       */
      logger.warn(CiiLogManager.formatException(e));
    }

    /* Error will not be logged at this level.
     * Stack will also contain entries from the ex.runProductionFileOpen();
     */
    try {
      ex.runProductionFileOpen();
    } catch (CiiException e) {

      /*
       * Access individual information
       */
      System.out.println(e.getTypeName()); //errorTypeID
      System.out.println(e.getCreationDate()); //detectionTimestamp
      System.out.println(e.getStackTraceAsString());//localization information
      System.out.println(e.getMessage()); //message


      /* This will exit the application with CiiException as the exception
       * is thrown from the main method.
       */

      throw e;
    }

  }

  /**
   * Methods on ErrorHandlingExample present the 2 level of exception
   * handling.
   */
  public void runProductionArrayElementCode() {

    /*
     * Error from lower level is handled as non-critical and logged
     * as warning.
     * The super-type CiiException also catches CiiArrayOutOfRangeException
     */
    ProductionCode pcHandled = new ProductionCode();
    try {
      pcHandled.codePartThatGetsArrayElement();
    } catch (CiiException ex) {
      logger.warn(CiiLogManager.formatException(ex));
    }

    /*
     * Error from lower level is handled but passed also to higher level.
     */
    ProductionCode pc = new ProductionCode();
    try {
      pc.codePartThatGetsArrayElement();
    } catch (CiiBadIndexException ex) {
      /*
       * Stack will contain details form CiiBadIndexException
       * Error will be propagated to upper level
       */
      throw ex;
    }
  }

  public void runProductionFileOpen() {
    /*
     * Non handled, error will be handled in the higher level.
     */
    ProductionCode pcNonHandled = new ProductionCode();
    pcNonHandled.codePartThatTriesToOpenFile();
  }

  private class ProductionCode {

    /**
     * Application level 3 error handling
     */
    public void codePartThatGetsArrayElement() {

      final ArrayList<Integer> testList =
            new ArrayList(Arrays.asList(1, 2, 3, 4, 5));
      final int arrayIndex = 9;

      /*
       * Example of array out of range error, where java native exception
       * is wrapped into
       * CiiArrayOutOfRangeException. The error is propagated to higher level.
       */

      try {
        testList.get(arrayIndex);
      } catch (IndexOutOfBoundsException ex) {
        /*
         * Internal exception will be caught, but nothing will be passed.
         * IndexOutOfBoundsException is native java exception.
         */
      }

      try {
        testList.get(arrayIndex);
      } catch (IndexOutOfBoundsException ex) {
        /* Internal exception will be caught, and passed
         * as a CiiBadIndexException
         *
         * Error message example:  "Index 9 is out of bounds!
         * Error will be propagated to higher level.
         */
        throw new CiiBadIndexException(arrayIndex, ex);
      }
    }


    public void codePartThatTriesToOpenFile() {

      /*
       * Non existing file is logged as info.
       */
      final String path = "data.txt";
      final File myDatafile = new File(path);
      if (!myDatafile.exists()) {
        /*
         * We treat this as non - error, we only log the information
         * to logger.
         */
        logger.log(Level.INFO, "File %s could not be opened", path);
      }

      /*
       * Non existing file is threaded as error and passed to higher level.
       */
      if (!myDatafile.exists()) {
        /*
         * Pass the exception to higher level.
         * Exception will have path added to the message.
         */
        throw new CiiFileHandlingException(path);
      }
    }
  }

Presented example will produce the following log messages:

14:53:41.056 [main] INFO ErrorHandlingUsageExample - Index 9 is out of bounds!, ErrorTypeName=elt.error.CiiBadIndexException, ErrorDateTime=1579445621042, ErrorMessage=Index 9 is out of bounds!, ErrorTrace=elt.error.CiiBadIndexException
     at elt.error.ErrorHandlingExample$ProductionCode.codePartThatGetsArrayElement(ErrorHandlingExample.java:127)
     at elt.error.ErrorHandlingExample.runProductionArrayElementCode(ErrorHandlingExample.java:75)
     at elt.error.ErrorHandlingExample.main(ErrorHandlingExample.java:41)
Caused by: java.lang.IndexOutOfBoundsException: Index: 9, Size: 5
     at java.util.ArrayList.rangeCheck(ArrayList.java:657)
     at java.util.ArrayList.get(ArrayList.java:433)
     at elt.error.ErrorHandlingExample$ProductionCode.codePartThatGetsArrayElement(ErrorHandlingExample.java:120)
     ... 2 more
,
14:53:41.069 [main] WARN ErrorHandlingUsageExample - File data.txt could not be opened
Exception in thread "main" elt.error.CiiFileHandlingException: Error while handling: data.txt
     at elt.error.ErrorHandlingExample$ProductionCode.codePartThatTriesToOpenFile(ErrorHandlingExample.java:154)
     at elt.error.ErrorHandlingExample.runProductionFileOpen(ErrorHandlingExample.java:90)
     at elt.error.ErrorHandlingExample.main(ErrorHandlingExample.java:49)

Source code of this example can be found in the client-api module under elt-common/java/common/test/elt/error/ErrorHandlingExample.java

8.4.6.1. Building an Error Stack¶

The following examples show sample code of building the exception stack in different languages. The exception stack is built by nesting the exceptions:

8.4.6.1.1. Building Error Stack in Java¶

try {
     try {
             throw CiiOldbDpExistsException("testDPUri");
             } catch (CiiOldbDpExistsException ex) {
                     throw (CiiBadIndexException(10, ex));
             }
     } catch (CiiBadIndexException ex) {
             ex.getCiiExceptionStackAsString();
     }
}

8.4.6.1.2. Building Error Stack in Python¶

try:
    try:
        raise CiiInvalidTypeException("")
    except CiiException as x:
        raise CiiBadIndexException(x, 10)
except CiiException as cix:
        print(cix.get_stack_trace())

8.4.6.1.3. Building Error Stack in CPP¶

try
{
  try
  {
    throw elt::oldb::CiiOldbDpExistsException("testDPUri");
  } catch (const elt::error::CiiOldbException& e) {
    // build exception stack of nested exceptions
    // CiiOldbDpExistsException is nested in CiiBadIndexException
    CII_THROW_WITH_NESTED(elt::error::CiiBadIndexException, e, 10);
  }
} catch (const elt::error::CiiException& e) {
  // flatten all nested exceptions and dump to string
  std::string nestedDetails = e.dumpWithNested();
}

8.4.6.2. Throwing a CPP Exception with Extra Information¶

CPP API provides additional macros which add line number and code filename to the error messages. Section A.2 in Appendix A contains the details.

The following example shows throwing a CPP exception with details (line number, function, file name, etc.). Standard CPP exception what() method (inherited from std::exception) shall be used to obtain the exception message. CPP API also provides the extra Dump() method (see details in section A.1).

try
{
  CII_THROW(elt::error::CiiFileHandlingException, "myfileName");
}
catch(const elt::error::CiiException& e)
{
  std::cout << e.What(); //error message
  std::cout << e.GetDetails(); //error details
  std::cout << e.Dump(); //error dump
}

8.4.6.3. Exception Serialization from CPP to Java¶

The following code shows an example of transferring the exception from CPP to Java. All the exceptions are serialized using the MAL CiiSerializableException.

//CPP method with CiiException on the sender side
std::string methodWithException(const std::string &par1) override
{
  throw elt::error::CiiInvalidDataTypeException(par1);
  return par1;
}

//Java - Receiving exception on the reciever side
try {
  final String unexpectedReply = serialzation.methodWithException(“Test”);
} catch (CiiSerializableException e) {
  logger.info(String.format("Type: %s", e.getCiiType()));
  logger.info(String.format("Host name: %s", e.getCiiHostName()));
  logger.info(String.format("File name: %s", e.getCiiFileName()));
  logger.info(String.format("Class name: %s", e.getCiiClassName()));
  logger.info(String.format("Method name: %s", e.getCiiMethodName()));
  logger.info(String.format("Message: %s", e.getCiiMessage()));
  logger.info(String.format("Creation Date: %d", e.getCiiCreationDate()));
  logger.info(String.format("Stack trace: %s", E.getCiiStackTrace()));
  logger.info(String.format("Exception stack: %s", e.getCiiExceptionStack()));
  logger.info(String.format("Details: %s", e.getCiiDetails()));
}

8.4.7. Using the Condition API¶

The following example shows the use of the condition API. Since the use is similar for Java, CPP and Python, only a Java example is presented.

public static void main(String[] args) {
  // Enable condition API for the process.
  CiiConditions.setEnabled(true);

  ErrorConditionsExample ex = new ErrorConditionsExample();

  try{
     ex.setRotation(250);
  } catch (CiiException x){
     System.out.println(x.getMessage());
  }
  ex.getListValue(10);
}

// get value from array of float
public float getListValue(int index){
  // check is element is in range.
   CiiConditions.checkElementIndexRange(index, list.length,
                new CiiBadIndexException(index));
   return list[index];
}

public void setRotation(int degrees){
  //pre condition check -- check if rotation in smaller than 360 deg.
  CiiConditions.checkArgument(degrees <= 360, new CiiOverRotatedException());
  axis.setRotation(degrees)
  axis.rotate()
  final int readRotation = 200;
  // post condition check -- check if axis is properly rotated.
  CiiConditions.checkArgument(readRotation == degrees,
      new CiiNotProperlyRotatedException());
}

8.5. Using the QT Error Dialog Widget¶

To use the error dialog widget, CiiExceptionDialog.hpp must be included into the source code. The dialog is located under the elt::error::gui namespace. The CiiExceptionDialog.hpp derives from the QDialog. The method exec() is therefore directly inherited from this QDialog class.

Exception dialog constructor takes a CiiException as an input parameter. All exception details are automatically parsed into the dialog, based on the CiiException type.

To display the widget, exec() method must be called. The following code shows the typical usage:

#include <CiiExceptionDialog.hpp>

try {
  throw elt::error::CiiInvalidTypeException();
} catch (const elt::error::CiiException& e) {
  gui::CiiExceptionDialog w(e);
  w.exec();

}

Figure 5‑1 shows an example of the Error widget dialog screen output. The widget displays all the important error information: Exception class (error type), exception message, time of the occurrence, thread ID, process ID, host, line number and stack trace.

Figure 5‑1 Error widget dialog screen output

8.6. Indexing service¶

Error indexer (2.2.2) is a tool that collects the data of exceptions and stores the exceptions data to Elasticsearch index. Using Elasticsearch for searching the exceptions provides structured and powerful search. Data provided by indexing service is used by the search script (4.5), when using the ElasticSearch option. As ElasticSearch returns data in JSON format, the data returned by the queries can be used for fast integration with other services.

8.6.1. Configuring indexing service¶

Error indexer uses configuration localDB for setting up the needed configuration. The configuration instance file must be created and deployed for error indexer to work properly. Use the following template to create the config file:

config:
  instance:
    __comment__: Configuration for error-indexer
    data:
      "@type": CiiErrorIndexerConfigClass
      fields:
        elasticSearchServiceAddress:
           "@type": MdString
           metadataInstance: ConfStringStd
           value: "localhost"
        elasticSearchIndexName:
           "@type": MdString
           metadataInstance: ConfStringStd
           value: "errors"
        centralErrorCodesRepositoryHost:
           "@type": MdString
           metadataInstance: ConfStringStd
           value: "https://svnhq8.hq.eso.org"
        centralErrorCodesRepositoryPaths:
           "@type": MdStringArray
           metadataInstance: ConfStringArrayStd
           value:
             - "p8/trunk/CONTROL-SYSTEMS/CentralControlSystem/CII/CODE/srv-error-indexer/sampleErrorCodesRepo/"
        lastProcessedRevision:
           "@type": MdInt32
           metadataInstance: ConfInt32Std
           value: 0
        centralErrorCodesPassword:
           "@type": MdString
           metadataInstance: ConfStringStd
           value: "myPass"
        centralErrorCodesUserName:
           "@type": MdString
           metadataInstance: ConfStringStd
           value: "myUsername"

After the file is created (we assume it is saved under ErrorIndexerConfig.cdi.yaml), it has to be deployed to the localDB. If localDB is not initialized, it should be initialized first.

config-tool init #LocalDb initialization.
config-tool deploy -i localDbConfiguration/ErrorIndexerConfig.cdi.yaml -a cii.config://services/CiiErrorIndexer

Following properties are set:

elasticSearchServiceAddress: Address of the elastic search server.
elasticSearchIndexName: Index name, which will hold the indexed data.
centralErrorCodesRepositoryHost: SVN hostname of central repository.
centralErrorCodesRepositoryPaths: Array of paths in the repository. Indexer will look for exception code under these subfolders.
lastProcessedRevision: Internally, last processed revision is stored. The setting should initially be set to 0.
centralErrorCodesPassword: password credential for SVN.
centralErrorCodesUserName: user name credential for SVN.

8.6.2. Running the indexing service¶

To manually run the indexer, execute the following command:

error-indexer

It is proposed to run the indexer every 30 minutes. A Linux time-based utility cron shall be used for this. To define the schedule to use the cron, put the following line into the /etc/crontab (change the {INTROOT} to proper location of the CII root installation folder – varriable $INTOOT cannot be used here):

30 * * * * {INTROOT}/bin/error-indexer

Another option (not documented in this manual) is to hook running of indexer on every SVN update.

8.6.3. Accessing the indexed data¶

Elasticsearch provides more powerful search capabilities of the exception definitions and more structured output searched data. As the data is returned in JSON format, it is more suitable for integration with other services, for example logging. Elasticsearch uses REST API to access the data. Any REST client can be used to access the errors data. For the following examples we will use curl.

Displaying all the errors in the index:

curl -XPOST 'http://ciielastichost:9200/errors/_search?pretty'

Elasticsearch provides Query DSL language [4], with powerful search capabilities. The following fields can be used, when searching the error index:

folder,
author,
namespace,
description,
language,
message,
version,
revision,
timestamp.

Elasticsearch supports different types of searches. When using the wildcard type of search, only lower case is supported (ElasticSearch standard analyzer is used [5]). Data in index can be in camel case, but the search will only work if the lower-case words are used in the search query. The search will not distinguish between lower and upper case.

Getting the error data based on the exception name:

curl -XPOST 'http://ciielastichost:9200/errors/_search?pretty' -d '{"query":{"match":  {"name": " CiiFileHandlingException"}}}' -H "Content-Type: application/json"

Wildcard search based on exception name:

curl -XPOST 'http://ciielastichost:9200/errors/_search?pretty' -d '{"query":{"wildcard":  {"name": "ciifile*"}}}' -H "Content-Type: application/json"

Searching the errors with the desired description:

curl -XPOST 'http://ciielastichost:9200/errors/_search?pretty' -d '{"query":{"match":  {"description": "This is the description"}}}' -H "Content-Type: application/json"

Searching the errors with the desired description and name:

curl -XPOST 'http://ciielastichost:9200/errors/_search?pretty' -d '{"query": {"bool": {"should": [{ "match": { "description": "This is the description"}}, { "match": { "message": "Error while"}}]}}}' -H "Content-Type: application/json"

Wildcard search with the desired description and name:

curl -XPOST 'http://ciielastichost:9200/errors/_search?pretty' -d '{"query": {"bool": {"should": [{ "wildcard": { "description": "* is the description"}}, { "wildcard": { "message": "*while"}}]}}}' -H "Content-Type: application/json"

Search with the desired language and name:

curl -XPOST 'http://ciielastichost:9200/errors/_search?pretty' -d '{"query": {"bool": {"must": [{ "match": { "language": "JAVA"}}, { "match": { "name": " CiiFileHandlingException"}}]}}}' -H "Content-Type: application/json"

Return the last record of wildcard search sorted by timestamp:

curl -XPOST 'http://ciielastichost:9200/errors/_search?pretty' -d '{"query":{"wildcard":  {"name": "ciifile*"}},"sort": [ { "timestamp": { "order": "desc" } } ], "size": 1}' -H "Content-Type: application/json"

8.7. API¶

8.7.1. CII Exception methods¶

CII-specific fields and methods are added to CiiException to provide the required functionalities:

stack trace: (execution stack trace - back trace) is provided by language-specific trace mechanisms in Java and Python. CPP uses the boost libbacktrace module. The getStackTraceAsString method provides a stack trace in a string format.

creation date: creation date/time is implicitly set by CiiException. The date/time is stored in milliseconds since Epoch. The getCreationDate method will return the number.

type name: type name is implicitly set by the exception definition. The getCiiType method will provide the type of the exception class.

The CPP CiiException class provides additional functionalities:

The following fields are provided as member fields with getter and setter methods:

threadId

lineNumber

functionName

filename

processId

hostname

details: additional exception trace details can be added to CPP exceptions. This attribute should be explicitly set by the developer. The getDetails and setDetails methods should be used to retrieve and store the information.

The two additional methods are added:

dump: method that returns string formatted information about all exception details and values.

dumpWithNested: method that returns string formatted information about all nested exceptions details and values.

8.7.2. CPP macros¶

CPP API provides macros which add details (file, function and line number) to the error messages. If these macros are not used, the exceptions will not contain the mentioned fields. The following macros are provided:

CII_THROW(exceptionType_t, …)
Throws the desired exception with additional information. First parameter is the exception type. All other parameters are variadic, and depend on the number of parameters the exception class defines.

CII_THROW_WITH_NESTED (exceptionType_t, nested_exception, …)
Throws the desired exception with additional information and nested exception. The first parameter is the exception type and the second parameter is the exception to be nested. All other parameters are variadic, and depend on the number of parameters the exception class defines.

8.7.3. Conditions API¶

The conditions API provides static methods for verifying arguments, null values, states and valid index positions in lists and arrays. It provides the following methods:

setEnabled(bool enabled)

getEnabled()

Methods for enabling and disabling the Conditions API. If Conditions API is disabled, checks will not be executed. The getEnabled method will return the current state of the enabled flag.

setFailFastMode(bool enabled)

getFailFastMode()

Methods for enabling and disabling the fail-fast mode. If fail-fast is enabled, failing the condition will cause the application to exit with an error. The getFailFastMode method will return the current state of the enabled flag.

checkArgument(bool expr, Exception ex)
Checks that the parameter expr is true. It can be used for validating arguments in methods. If the expression evaluates to false, the given exception is thrown.

checkNotNull(T value, Exception ex)
Checks that the value is not null. It returns the value directly, so you can use checkNotNull(value, ex) inline. In case of null, the given type exception is thrown.

checkState(boolean value, Exception ex)
Checks some state of the object, not dependent on the method arguments. For example, an iterator might use this to check that method next() has been called before any call to remove() method. If the expression evaluates to false, the given exception is thrown.

checkElementIndexRange (int index, int size, Exception ex)
Checks that index is a valid element index in a list, string, or array with the specified size. An element index may range from 0 inclusive to size exclusive. Instead of passing the whole list, string, or array, only it’s size is passed as parameter. If the index exceeds the size, the given exception is thrown.

The condition API for specific languages (Java, CPP and Python) is shown in the following sections.

8.7.3.1. Java¶

public class CiiConditions {
    public static boolean getEnabled() {...}

    public static void setEnabled(boolean enabled) {...}

    public static boolean getFailFastMode(){...}

    public static void setFailFastMode(boolean failFastMode) {...}

    public static <E extends Exception> void checkArgument(boolean expr, E ex) throws E {...}

    public static <T, E extends Exception> void checkNotNull(T value, E ex) throws E {...}

    public static <E extends Exception> void checkState(boolean value, E ex) throws E {...}

    public static <E extends Exception> void checkElementIndexRange(int index, int size, E ex) throws E {...}

}

8.7.3.2. CPP¶

class CiiConditions {
public:

    static bool GetEnabled();

    static void SetEnabled(bool enabled);

    static bool GetFailFastMode();

    static void SetFailFastMode(bool failFast);

    template<typename E, class T, typename ... Args>
    static void CheckArgument(bool expr, Args ... args);

    template<typename E, class T,  typename ... Args>
    static void CheckNotNull(T value, Args ... args)

    template<typename E, class T, typename ... Args>
    static void CheckState(T value, Args ... args)

    template<typename E, class T, typename ... Args>
    static void CheckElementIndexRange(size_t index,
                 size_t size, Args ... args);

};

8.7.3.3. Python¶

class CiiConditions:

   @staticmethod
  def get_enabled():
      pass

  @staticmethod
  def set_enabled(enabled):
      pass

  @staticmethod
  def get_fail_fast_mode():
      pass

  @staticmethod
  def set_fail_fast_mode(failFastMode):
      pass

  @staticmethod
  def check_argument(expr, ciiException):
      pass

  @staticmethod
  def check_not_null(value, ciiException):
      pass

  @staticmethod
  def check_state(expr, ciiException):
      pass

  @staticmethod
  def check_element_index_range(index, size, ciiException):
      pass

8.8. Serialization code example¶

The following code presents example of serialization of exceptions over MAL. The example consists of MAL CPP server and Java client. Examples for other languages can be found in the client-api module source code:

Java: client-api/elt-common/java/errorNonUnitTests
CPP: client-api/elt-common/cpp/error/errorSerialClnt, errorSerialSrv
Python: client-api/elt-common/runScripts/testZpbClientPythonException, testZpbServerPythonException

8.8.1. CPP serialization server code¶

#include <CiiSerializableException.hpp>
#include <CiiSerializationTest.hpp>
#include <CiiInvalidDataTypeException.hpp>

#include <mal/rr/qos/QoS.hpp>
#include <mal/Cii.hpp>
#include <mal/Mal.hpp>
#include <mal/utility/LoadMal.hpp>
#include <mal/rr/ServerContextProvider.hpp>
#include <mal/rr/ServerContext.hpp>
#include <mal/rr/ServerAmi.hpp>
#include <mal/rr/RrEntity.hpp>

#include <stdexcept>
#include <memory>
#include <iostream>
#include <boost/thread/future.hpp>

namespace {

const std::string EXCEPTION_CALL = "error";

/**
 * Simulate some work
 */
static void simulateWork() {
  std::this_thread::sleep_for(std::chrono::seconds(2));
}

class CiiSerializationTestImpl : public virtual elt::error::icdTest::CiiSerializationTest
{
  std::string methodWithException(const std::string &par1) override
  {
    std::cout << "Got exception call with parameter: " << par1 << std::endl;
    if (par1 == EXCEPTION_CALL)
    {
      elt::mal::throw_exception(elt::error::CiiInvalidDataTypeException(par1));
    }
    return par1;
  }

  std::shared_ptr<::elt::mal::rr::Ami<std::string>>
  amiMethodWithException(const std::string& par1)
  {
    // Obtain ptr to ServerAmi
    std::shared_ptr<::elt::mal::rr::Ami<std::string>> ami = elt::mal::rr::ServerContextProvider::
        getInstance<elt::mal::rr::ServerContext<std::string>>().createAmi();

    std::cout << "Ami call with param: " << par1 << std::endl;

    if (par1 != EXCEPTION_CALL)
    {
      auto future = boost::async(boost::launch::async, [=]() {
        simulateWork();
        // first response
        ami->complete("ONE");

        simulateWork();
        // second response
        ami->complete("TWO");

        simulateWork();
        // third response
        ami->completed("THREE");
      });
    }
    else
    {
      auto future = boost::async(boost::launch::async, [=]() {
        simulateWork();
        // first response
        ami->complete("ONE");

        simulateWork();
        // second response
        ::elt::error::CiiInvalidDataTypeException e = elt::error::CiiInvalidDataTypeException(par1);
        e.setFileName(boost::filesystem::path(__FILE__).filename().string());
        e.setFunctionName(__FUNCTION__);
        e.setLineNumber(__LINE__);
        e.setClassName(boost::core::demangle(typeid(*this).name()));
        ami->completeExceptionally(e);

        simulateWork();
        // third response
        ami->completed("THREE");
      });
    }
    return ami;
  }
};

} // namespace

int main(int ac, char* av[]) {
  std::cout << "CPP ZPB server test start." << std::endl;

  // Load DDS MAL mapping
  std::shared_ptr<::elt::mal::Mal> ddsMal = elt::mal::loadMal("zpb", {{"dds.domain", "100"}});
  ::elt::mal::CiiFactory &factory = elt::mal::CiiFactory::getInstance();
  factory.registerMal("zpb", ddsMal);

  elt::mal::Uri uri("zpb.rr://0.0.0.0:12081/CiiSerializationTest");

  try {
    // Create server, default QoS.
    std::unique_ptr<elt::mal::rr::Server> server = factory.createServer(uri, elt::mal::rr::qos::QoS::DEFAULT, {});

    // Register the service.
    std::shared_ptr<elt::mal::rr::RrEntity> exSer(new CiiSerializationTestImpl());
    server->registerService<elt::error::icdTest::CiiSerializationTest, true>("inst_1", exSer);

    // Process commands until server->shutdown() is called.
    server->run();
  } catch (std::exception& exc) {
    std::cerr << "EXCEPTION: " << exc.what() << std::endl;
  }

  std::cout << "Server stopped." << std::endl;
  return 0;
}

8.8.2. Java serialization client code¶

import java.net.URI;
import java.util.Properties;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import org.apache.logging.log4j.Logger;
import elt.error.icd.CiiSerializableException;
import elt.error.icdTest.CiiSerializationTestSync;
import elt.log.CiiLogManager;
import elt.mal.CiiFactory;
import elt.mal.Mal;
import elt.mal.rr.Ami;
import elt.mal.rr.qos.QoS;
import elt.mal.rr.qos.ReplyTime;
import elt.mal.zpb.ZpbMal;

public class ExceptionSerializationClientTest {

  private static final Logger logger = CiiLogManager.getLogger(ExceptionSerializationClientTest.class.getName());
  private static final String NO_EX_CALL = "first";
  private static StringBuilder outData = new StringBuilder();

  public static void main(String[] args) {
    logger.debug("Java test zpb client started.");

    final CiiFactory factory = CiiFactory.createInstance();

    final URI uri = URI.create("zpb.rr://127.0.0.1:12081/CiiSerializationTest/inst_1");
    logger.debug("URI: " + uri.toString());

    Mal mal = new ZpbMal();
    factory.registerMal("zpb", mal);

    try (CiiSerializationTestSync serialzation =
        factory.getClient(uri, new QoS[] {new ReplyTime(3, TimeUnit.SECONDS)}, new Properties(),
            CiiSerializationTestSync.class)) {

      logger.info("Perfomrming simple call test.");
      try {
        final String reply = serialzation.methodWithException(NO_EX_CALL);
        if (!reply.equals(NO_EX_CALL)) {
          logger.error("Error!!!! Did not get expected string.");
          System.exit(1);
        }
        final String unexpectedReply =
            serialzation.methodWithException(ExceptionSerializationServerTest.EXCEPTION_CALL);
        logger.error("Error!!!! Should not be here.");
      } catch (CiiSerializableException e) {
        final CiiException ex = CiiException.fromSerializable(e);
        logger.debug("Got CiiSerializableException.");
        logger.info(String.format("Type: %s", ex.getCiiType()));
        logger.info(String.format("Host name: %s", ex.getCiiHostName()));
        logger.info(String.format("File name: %s", ex.getCiiFileName()));
        logger.info(String.format("Class name: %s", ex.getCiiClassName()));
        logger.info(String.format("Method name: %s", ex.getCiiMethodName()));
        logger.info(String.format("Message: %s", ex.getCiiMessage()));
        logger.info(String.format("Creation Date: %d", ex.getCiiCreationDate()));
        logger.info(String.format("Stack trace: %s", ex.getCiiStackTrace()));
        logger.info(String.format("Exception stack: %s", ex.getCiiExceptionStackAsString()));
        logger.info(String.format("Details: %s", ex.getCiiDetails()));

        //the data to be checked by the integration test
        outData.append("Type -- ").append(e.getCiiType()).append(";");
        outData.append("Message -- ").append(e.getCiiMessage()).append(";");
        outData.append("Exception stack -- ").append(ex.getCiiExceptionStackAsString());

      }

      logger.info("Perfomrming Ami call test.");
      try {
        logger.info("OK call.");
        final Ami<String> okTasks = serialzation.amiMethodWithException(NO_EX_CALL);
        for (CompletableFuture<String> reply : okTasks) {
          final String result = reply.get();
          logger.info("Reply: " + result);
          logger.info("Task complete: " + okTasks.isDone());
        }

        logger.info("Error call.");
        final Ami<String> errorTasks = serialzation.amiMethodWithException(ExceptionSerializationServerTest.EXCEPTION_CALL);
        for (CompletableFuture<String> reply : errorTasks) {
          final String result = reply.get();
          logger.info("Reply: " + result);
          logger.info("Task complete: " + errorTasks.isDone());
        }
        logger.error("(Ami) Error!!!! Should not be here.");
      } catch (InterruptedException e) {
        logger.error("Unexpected error: ", e);
      } catch (ExecutionException e) {
        final Throwable th = e.getCause();
        if (th instanceof CiiSerializableException) {
          final CiiSerializableException se = (CiiSerializableException) th;
          final CiiException ex = CiiException.fromSerializable(se);
          logger.debug("Got CiiSerializableException.");
          logger.info(String.format("Type: %s", ex.getCiiType()));
          logger.info(String.format("Host name: %s", ex.getCiiHostName()));
          logger.info(String.format("File name: %s", ex.getCiiFileName()));
          logger.info(String.format("Class name: %s", ex.getCiiClassName()));
          logger.info(String.format("Method name: %s", ex.getCiiMethodName()));
          logger.info(String.format("Message: %s", ex.getCiiMessage()));
          logger.info(String.format("Creation Date: %d", ex.getCiiCreationDate()));
          logger.info(String.format("Stack trace: %s", ex.getCiiStackTrace()));
          logger.info(String.format("Exception stack: %s", ex.getCiiExceptionStackAsString()));
          logger.info(String.format("Details: %s", ex.getCiiDetails()));
        } else {
          logger.error("Unexpected error: ", e);
        }
      } catch (Throwable e) {
        logger.error("Unexpected throwable: ", e);
      }

    } finally {
      mal.close();
      factory.close();
    }
    logger.debug("Client done.");
    System.out.print(outData.toString());
    System.out.print("PASS");
  }

}