There exists one test project for each cultivate project. Additionally you need to checkout the cultivate.test project, which contains common files for all tests. The following directories may exist in each project:

• plain-tests: Contains plain JUnit Java tests.
• plugin-tests: Contains JUnit tests that need an eclipse environment to run.
• prolog-tests: Contains tests for prolog predicates, also based on JUnit.

For writing prolog tests take a look at the MisnamedTestCaseTest example in cultivate.basic.test:

Creation of a new test consists of 4 steps:

1. Write the test data
2. Generate the factbases
3. Write the test
4. Run the test

First you need to write your testdata. Do this by creating a new testdata-* src folder and adding it to the build path. Now write your packages and classes. Afterwards you must run the generator-factbases-project.launch via the eclipse launcher menu. This will create the file *.qlf in the factbases directory. Now you can write your test by extending the AbstractPrologTestCase. It will automatically consult the prolog file prolog/load.pl of your project. Additionally you have the following helper methods to use:

• consult(String filename): Consults additional prolog files given by filename.
• consultResource(String filename): Consult prolog file in the current source-folder's resources folder
• useTestData(String testdataName): Consults the factbase given by factbaseName. Note: If the directory is called testdata-123 you only have to give 123 as testdataName.
• consultJTFacts(): Will cronsult the JTransformer facts
• callPredicate(String query): This will call a predicate and return all results as List.
• assertResultSize(int size): Assert the number of results of the last called predicate.
• setExpectedResults(String key, String… values): Sets the expected results for the given key. Can be called more than once with different keys. Tuples of values from each call will match then.
• assertExpectedResultsExist: Asserts the expected result tuples have been returned by the predicate call.
• assertPredicateReturnsFalse: Asserts that the predicate has an empty result set
• assertPredicateReturnsTrue: Asserts that the predicate has a non-empty result set

• getIdOfFullQualifiedName: Returns the ID for a qualified name
• fullFilename: Get the full filename of a (class)file in the sourcefolder
• getIdOfSourceLocation: Get the ID of an element described by the the full filename and the location of the element

• dumpResults: Writes a list to STDOUT of results of the last prolog query
• dumpExpectedResultSettings: Same as dumpResults with other output format

For executing predicates in a changing context, the ContextualPredicate class is provided. it supports the following methods:

• get/setContextName: to set the name of the context parameter in the predicate, defaults to Context
• setVariablesToCheck: Defines the names of the variables to verify
• assertResult: Asserts a result in a specific context. Note: If you execute several asserts in the same context, execute them in a row for speed-up.
• assertResultSize(String Context, int size): Assert the number of results of the predicate in the Context.
• dumpResult: see above
• dumpExpectedResultSettings: see above

IMPORTANT: Before you can run your tests with JUnit you have to alter cultivate.test/src/org.cs3.cultivate.test.framework/test.properties to match the correct paths.

• Stuff to test:
  1. Java parts
  2. Prolog parts

• General Criterias
  1. Independence of tests and code

• Evaluation Criterias for Prolog tests:
  1. Fast Tests
  2. Automated Tests
  3. Real classes as testdata
  4. Clear relations between testdata, tests and code
  5. Test cases should be easy to write
  6. Test framework should be easy to use

1. Java Parts
   • Plain Tests: Test of java cultivate core functionality
   • Plugin Tests: Test of cultivate functionality related to eclipse

1. Prolog Parts
   • Prolog Tests (based on plugin tests):
     1. Slow
     2. No automation because of a bug
        • plwin does not close if test fails
     3. Testdata are no real classes
     4. Testdata in the tests
     5. Assertions not intuitive to check
     6. Easy to use, since junit test integration is good in eclipse
     7. JTransformer predicates readily available for the tests.
     8. Same internal mechanisms as in Cultivate itself.
   • Fitnesse Tests
     1. Fast
     2. Automation works
     3. Real classes as test data
     4. Traceability between testdata and factbases not given
     5. Easy writable tests through wiki syntax
     6. Fitnesse is unintuitive for tests at the beginning
     7. JTransformer predicates need to be explicitly consulted.
   • JTransformer (based on plugin-tests)
     1. Slow
     2. Automation works
     3. Testdata are real classes, but zipped
     4. Testdata in the tests
     5. Assertions not intuitive to check
     6. Easy to use, since junit test integration is good in eclipse

• Assumptions:
  • One project for the test framework (cultivate.tests)
  • One test project per project to test (e.g. cultivate.core ⇒ cultivate.core.tests)

• Fitnesse
  • One sourcefolder per factbase in each test project and precompile and filter factbases based on sourceFolderT
  • Prepare Fixture that consults all JTransformer predicates.

• Junit Tests
  1. One sourcefolder per factbase in each test project
     • Still needs slow plugin tests
  2. additionaly precompile and filter factbases based on sourceFolderT
     • would get rid of plugin tests and move to plain prolog tests
  3. Improve assertion checking for prolog results
  4. Timeout for prolog tests, so that plwins get removed.

• Write a test environment for plain prolog tests
  • Java Prolog Queries not testable
  • Relation between factbase and source code not visible again