[cdwg] New Test Framework Development - Automation_framework_evaluation statements

[Roman Grigoryev]

Hi,
I would like to discuss some statements from
http://wiki.opensfs.org/Automation_framework_evaluation

Some items from "Some requirements/desires for the automation
framework" part  looks like out of test execution framework.

Item 'Should facilitate easy interpretation of test "trends"; i.e.
statistical-based test results' is really far from execution, and it is
part of reporting system. I prefer to avoid mixing execution and
reporting systems.

Item 'Collect test output status in a database looks too specific'. I
prefer to re-define this requirements in something like this "be able
to extend save test result in other formats"

Item 'Able to automatically start testing based on various triggers,
e.g. git commit hooks' looks too specific also. Git commit hooks just
execute command line. So, requesting "initiating start from command
line" is more simple. Which other triggers could be used exclude
command line?

Items 'Aware of multiple clusters' and 'Maintains a prioritizable job
queue both' mean execution framework with dedicated server where
multiple configuration and queues are kept and monitored. But with this
architecture we are loosing mobility. I think, there is mixed
requirements for testing framework and testing system for specific
needs. 

Thanks,
	Roman