Welcome To Our Shell

Mister Spy & Souheyl Bypass Shell

   Integration Test Concept for MIB_STD20
   -------------------------------------- 


The integration tests in the project MIB_STD20 are
organized hierarchical in the directory structure
underneath  AP_IntegrationTest .

   o   Each testclass is a subdirectory in this testtree with
       the belonging tests (or test-subclass) as subdirectories

   o   Each test is a subdirectory in this testtree with
       the belonging test-logs as subdirectories

All subdirectories in that testtree are named 'test items',
and a subdirectory of a test item is a 'child item'.



=====================================================

A.  Structure of the test tree
..............................


   The  item.py  file
   ------------------

For reporting purposes, each test item should contain a file
item.py  (together with an empty file  __init__.py)
which has the only one class  cls  with the method:

   self.result():

which returns the string:


:RESULT:      <passed>/<warning>/<error>/<notpassed>
:REMARK:      <shortdut description>
:USER:        <user information - date, time, account>
:ALLREQU:     100


and further optional  Requirements  in form of:

:Requ1:    <test execution needed>
:Requ2:    <test execution needed>
....




=====================================================


   The standard item.py
   --------------------

There is a standard item.py  which can
be taken as an   item.py  for a test item.

This special module has further functions:

   self.run():

      Generates a child item (a subdirectory),
      which also contains an  item.py . The
      originary run method is self-reproduced.


The  item.py  of the generated child item
inherits all methods of all ancestor item.py's



=====================================================

   The caller script pytest
   ------------------------

To call the methods of a test item, there is a caller
script  pytest . This is a python script taking as the
first parameter the bewished method, the last parameter
is the relative path to the test item, and all other
parameters.

So if you call:

    python3 <path-to>/pytest r(un) INTEG

from the directory AP_IntegrationTest, a child item
of INTEG will be generated.

If the last parameter does not indicate a test item,
the actual test item will be chosen

For the reason of inheritance of the ancestor-item.py's,
pytest goes up and up the test tree until it is able to
execute the test item without an error


Hint: It is recommended to prepare a file y.bat with
the following line:

   python3 <path-to>/pytest %1 %2 %3 %4 %5 %6 %7 %8

and put it into the path, so that you can say for
the above command:

   y r(un) INTEG



=====================================================

   User-defined  item.py's   with run method using sync()
   ------------------------------------------------------

It is recommended to overload the run method of
generated  item.py's  to get special child items
which can be test logs or generated test cases.

To generate a child item inside a user-defined run method,
there is the method self.sync() in the standard item.py
which acts as following:

o) sync writes the child test item into a file.

o) The values of self._result, self._remark, self._user and self._requ
   will be entered in that file in the method  result() .

o) The content of self._log will be posted into a method  log  .

o) Moreover, if that is valid python code, it will be copied
   directly in that file.

o) If a  'jump function' (a function which has a name t<nn> and
   sync() is called from run or t<mm> where <nn>  >  <mm>) exists,
   self  will be dumped and be written as method  dump_data  into
   the child item file, for the purpose of later re-awake.



=======================================================

Result and Remark
-----------------

Before you sync(), the following variables should be set:

   self._result = <pass>/<warn>/<err>/<notp>
   self._remark = <text>

        where:

         <pass>:           number of passed tests
         <warn>:           number of passed tests with warning
         <err>:            number of errorneous tests
         <notp>:           number of not passed tests
         <text>:           free text

If <text> = "SLEEP<n>", then the test item process will
have a 'sleep time' of <n> seconds from dumping time.
Attemps to re-awake it BEFORE that time will fail.

If you give  sync()  a parameter, it will be set to self._remark
before syncing the test item.


=====================================================


The next_test method
--------------------

By default, the numbering of the child items goes as:

    t001, t002, t003 .....

This is ruled out by the method  next_test  which can be
overloaded. Each child item inherits the original
next_test0  method from  the standard item.py


=====================================================



Re-awakening and proceding a synced test item
----------------------------------------------

To re-awake a test item process it inherits
from the standard item.py the method  proceed:

   y  proceed  [<test-item>]    or

   y  c        [<test-item>]


It will resume the status of the dumped test process
and proceed to execute it. If there is again
a 'jump function', a sync() call in the actual function
will generate dump information again, until
there is no dump function anymore.

You can choose the 'jump function' manually by
setting  self.___jump___  to that method name



=====================================================

USE CASE 1: Manual Tests
========================

A manual test contains an   item.py  with the
run method  run , which only does the following:

  o  It sets   self._log  to a certain text which
     is the description of what to do for the tester

  o  It sets   self._remark  to  'SLEEP99999999'
     (n 'infinite' sleep time)

  o  It syncs the test run, e.g. it will be stored
     into the file system

So we have for the run method:

   def run(self,pars):

      self._log = '''
Check whether the system is consistent.
RESULT:  ...   (OK/NOK)
'''
      self.sync("SLEEP99999999")


====================================================

Running that test item will produce a child item
as a subdirectory  t<nn> , which contains an  item.py
where the  self._log  content appears as the
return text of the method  self.log , which
can be MANUALLY manipulated.

Hence, the test should contain a further method,
t01, which is executed after re-awakening of the
sleeping test run. t01 should evaluate the manual
entries of  self._log  , perhaps by evaluating it
with regular expressions.

For example, it could be:

      
   def t01 (self):
      self._result = "0/0/1/0"
      self._remark = "ERROR. Test failed."
      self._log    = self._log + "TEST FINISHED."
      m = re.search(r"RESULT: +(OK|NOK)",self._log)
      if m:
         if m.group(1) == "OK":
            self._result = "1/0/0/0"
            self._remark = "Test successfully passed."
      self.sync()


====================================================

Practically:
(see example in INTEG/MIB_STD2_SITS_AudioManagement/Audio_Mute)

A)  Generate a new test in a directory MANUAL_TESTS
    with:

        y r(un)

B)  The created child, a directory, rename for example to
    TEST_ABC. Change the run method in the created
    TEST_ABC/ item.py to the above, and add the method
    t01 from above.

C)  Execute the test by going into   TEST_ABC
    and then:

        y r(un)

D)  In the created file t<nn>/item.py change
    the test description with the proper value,
    e.g.:
        RESULT:   ...    ->   RESULT:   OK
    or:
        RESULT:   ...    ->   RESULT:   NOK

E)  Change  SLEEP99999999  to  SLEEP0
    in the  remark  method  of t<nn>/item.py
    so that it can be re-awaked.

F)  Proceed the test run from  TEST_ABC

        y proceed  t<nn>   (or: y  c  t<nn>)

    or from TEST_ABC/t<nn> with:

        y proceed   (or:  y  c)


G)  Look again at the result TEST_ABC/t<nn>/item.py


====================================================

This workflow is a little bit much manual work,
so there is the ability to make it more automatic
by setting the environment variable  autoedit  to
your favorite editor (for example:
set autoedit=notepad). Then after starting the
test run it will automatically open that editor
and - after changing the text  - the test run will
be continued automatically.


An example is to be found in
INTEG/MIB_STD2_SITS_AudioManagement/Audio_Mute.
There the  self.-log  content  comes from a
inherited method  get_integ_text  which constructs
the proper test text from the name of the test module.


Keep attention, that all changes to the content of
the return values of the methods  log, result, remark
will change the values of self._log, self._result
and self._remark in the sleeping and then re-awaked
test run.


====================================================


USE CASE 2: Automatic tests
===========================

It is easy to transform a manual test into an automatic test:

The run method executes actions against the test object,
writes files, compiles, reads the database, and so on.

Then the run method should set proper values for
self.result and self.remark.

Nevertheless, an automatic test can have proceeding functions
t<nn>, too. Before sync the method before, set  self._result
to SLEEP<nnn>, where <nnn> is the bewished sleeping time.
It will not be possible to re-awake the test item before
the slepping time is expired.

Moreover, you can mix syncs in the test run with
normal sleep times and requesting sleeps.


====================================================

USE CASE 3: Generating tests with a test generator
--------------------------------------------------

A test generator also is a test item, e.g.
a directory in the test tree with a suitable item.py

Its run method shall contain the program code for
the generated test, so that we have to write it
like this:

   def run (self,pars):
   
      self._log = '''
   def run (self,pars):
      self._log = self.get_integ_test(self.__module__)
      self.sync("SLEEP99999999")
      
   def t01 (self):
      self._result = "0/0/1/0"
      self._remark = "ERROR. Test failed."
      self._log    = self._log + "TEST FINISHED."
      m = re.search(r"RESULT: +(OK|NOK)",self._log)
      if m:
         if m.group(1) == "OK":
            self._result = "1/0/0/0"
            self._remark = "Test successfully passed."
      self.sync()
'''
      self.sync()


====================================================

With a run of this test item we get a test which
gets its test description by its directory name
which is evaluated by the   get_integ_test  method.

The method   get_integ_test  in that example comes
by inheritance from a test item above in the
test tree hierarchy.


---------

The run method of a test-generator also can contain
values chosen by random.

---------

See example in INTEG/MIB_STD2_SITS_AudioManagement



====================================================

USE CASE 4: Fitting the next_test method to get
            certain named test cases
-----------------------------------------------

In Use case 3, we got a test which evaluates the
module name self.__module__ to get the certain
test text. Hence, it is necessary, to give the
generated test item the correct name. This can be
managed by renaming the created child item.

But a better method is, to overload the standard
next_test method by an own.

A next_test method takes a item-name,
and then generates the name of its decessor item.



====================================================

So we add the own next_test method to the test
generator of use case 3:

   def next_test (self,o):
      integ_cases = self.get_integ_test(re.sub("\.item$", ...
                            ".xxx.item",self.__module__))
      for x in (integ_cases):
         if   (o == x):
            o = ""
         elif (o == ""):
            return(x)
      return(o)


where the method  get_integ_test  gives all existing test cases
as an array (because the parameter given points to a non-existent
test, and get_integ_test is made to give in this case
exactly all existing tests). In the for loop the decessor item name
will be given, it there is one, other wise an empty string.


See again the example in INTEG/MIB_STD2_SITS_AudioManagement




====================================================

It remains to look at the method get_integ_test. In the example
above, it comes from  INTEG/item.py, and it reads the
csv-data which is stored beside the  item.py  in INTEG

Example INTEG/item.py also shows a generator, but for
test classes, with an own next_test method again.

The run method of  INTEG/item.py  again covers
the run method of its childs in an own run method
(see it in the file). So we have a coverage
of code twice in this run method, and so it is
necessary to mask the ''' in the second cover level.

This can be done by setting 'X'' for ''' in the
second level. The sync method will resolve this
again, and will make:

   'X''    to    '''
   'XX''   to    'X''

and so on. So you can cover code in the run method
in arbitrary many levels.



====================================================

Example: The complete workflow
------------------------------

So we have the followiung steps in the example above:

In test item  INTEG:

    y r     generates the test classes
            MIB_STD2_SITS_AudioManagement,
            MIB_STD2_SITS_DisplayManagement,
            .....

In the test classes: y r  generates the testcases
e.g. in MIB_STD2_SITS_AudioManagement:

    y r    generates the test cases
           Audio_Mute,
           Switch_Audio_Source,
           ...

In the test cases:  y r  execuctes a test and writes
its result in an test log item  t<nn>. It is necessary
to enter the test result in the dumped test run process
manually and to proceed and finish it.

This can be supported by setting the environment variable
autoedit, for example:  set autoedit=notepad