waf/docs/book/task_generators.txt

=== General purpose task generators

So far, various task generators uses have been demonstrated. This chapter provides a detailed description of task generator structure and usage.

==== Task generator definition

The chapter on make-like rules illustrated how the attribute 'rule' is processed. Then the chapter on name and extension-based file processing illustrated how the attribute 'source' is processed (in the absence of the rule attribute). To process 'any attribute', the following properties should hold:

. Attributes should be processed only when the task generator is set to generate the tasks (lazy processing)
. There is no list of authorized attributes (task generators may be extended by user scripts)
. Attribute processing should be controlable on a task generator instance basis (special rules for particular task generators)
. The extensions should be split into independent files (low coupling between the Waf tools)

Implementing such a system is a difficult problem which lead to the creation of very different designs:

. _A hierarchy of task generator subclasses_ It was abandoned due to the high coupling between the Waf tools: the C tools required knowledge from the D tool for building hybrid applications
. _Method decoration (creating linked lists of method calls)_ Replacing or disabling a method safely was no longer possible (addition-only), so this system disappeared quickly
. _Flat method and execution constraint declaration_ The concept is close to aspect-oriented programming and might scare programmers.

So far, the third design proved to be the most flexible and was kept. Here is how to define a task generator method:

[source,python]
---------------
top = '.'
out = 'build'

def configure(conf):
	pass

def build(bld):
	v = bld(myattr='Hello, world!')
	v.myattr = 'Hello, world!' <1>
	v.myMethod() <2>

from waflib import TaskGen

@TaskGen.taskgen_method <3>
def myMethod(tgen): <4>
	print(getattr(self, 'myattr', None)) <5>
---------------

<1> Attributes may be set by arguments or by accessing the object. It is set two times in this example.
<2> Call the task generator method explicitly
<3> Use a python decorator
<4> Task generator methods have a unique argument representing the current instance
<5> Process the attribute 'myattr' when present (the case in the example)

The output from the build will be the following:

[source,shishell]
---------------
$ waf distclean configure build
'distclean' finished successfully (0.001s)
'configure' finished successfully (0.001s)
Waf: Entering directory `/tmp/simpleproject/build'
hello world
Waf: Leaving directory `/tmp/simpleproject/build'
'build' finished successfully (0.003s)
---------------

NOTE: The method could be bound by using 'setattr' directly, like for binding any new method on a python class.

==== Executing the method during the build

So far, the task generator methods defined are only executed through explicit calls. Another decorator is necessary to have a task generator executed during the build phase automatically. Here is the updated example:

[source,python]
---------------
top = '.'
out = 'build'

def configure(conf):
	pass

def build(bld):
	bld(myattr='Hello, world!')

from waflib import TaskGen

@TaskGen.taskgen_method <1>
@TaskGen.feature('*') <2>
def methodName(self):
	print(getattr(self, 'myattr', None))
---------------

<1> Bind a method to the task generator class (redundant when other methods such as 'TaskGen.feature' are used)
<2> Bind the method to the symbol 'myfeature'

The execution results will be the following:

[source,shishell]
---------------
$ waf distclean configure build --zones=task_gen <1>
'distclean' finished successfully (0.004s)
'configure' finished successfully (0.001s)
Waf: Entering directory `/tmp/simpleproject/build'
23:03:44 task_gen posting objects (normal)
23:03:44 task_gen posting >task_gen '' of type task_gen defined in dir:///tmp/simpleproject> 139657958706768 <2>
23:03:44 task_gen -> exec_rule (139657958706768) <3>
23:03:44 task_gen -> process_source (139657958706768) <4>
23:03:44 task_gen -> methodName (139657958706768) <5>
Hello, world!
23:03:44 task_gen posted <6>
Waf: Leaving directory `/tmp/simpleproject/build'
23:03:44 task_gen posting objects (normal)
'build' finished successfully (0.004s)
---------------

<1> The debugging zone 'task_gen' is used to display the task generator methods being executed
<2> Display which task generator is being executed
<3> The method 'exec_rule' is used to process the 'rule'. It is always executed.
<4> The method 'process_source' is used to process the 'source' attribute. It is always executed exept if the method 'exec_rule' processes a 'rule' attribute
<5> Our task generator method is executed, and prints 'Hello, world!'
<6> The task generator methods have been executed, the task generator is marked as done (posted)

==== Task generator features

So far, the task generator methods we added were declared to be executed by all task generator instances. Limiting the execution to specific task generators requires the use of the 'feature' decorator:

[source,python]
---------------
top = '.'
out = 'build'

def configure(conf):
	pass

def build(bld):
	bld(features='ping')
	bld(features='ping pong')

from waflib import TaskGen

@TaskGen.feature('ping')
def ping(self):
	print('ping')

@TaskGen.feature('pong')
def pong(self):
	print('pong')
---------------

The execution output will be the following:

[source,shishell]
---------------
$ waf distclean configure build --zones=task_gen
'distclean' finished successfully (0.003s)
'configure' finished successfully (0.001s)
Waf: Entering directory `/tmp/simpleproject/build'
16:22:07 task_gen posting objects (normal)
16:22:07 task_gen posting <task_gen '' of type task_gen defined in dir:///tmp/simpleproject> 140631018237584
16:22:07 task_gen -> exec_rule (140631018237584)
16:22:07 task_gen -> process_source (140631018237584)
16:22:07 task_gen -> ping (140631018237584)
ping
16:22:07 task_gen posted
16:22:07 task_gen posting <task_gen '' of type task_gen defined in dir:///tmp/simpleproject> 140631018237776
16:22:07 task_gen -> exec_rule (140631018237776)
16:22:07 task_gen -> process_source (140631018237776)
16:22:07 task_gen -> pong (140631018237776)
pong
16:22:07 task_gen -> ping (140631018237776)
ping
16:22:07 task_gen posted
Waf: Leaving directory `/tmp/simpleproject/build'
16:22:07 task_gen posting objects (normal)
'build' finished successfully (0.005s)
---------------

WARNING: Although the task generator instances are processed in order, the task generator method execution requires a specific declaration for the order of execution. Here, the method 'pong' is executed before the method 'ping'

==== Task generator method execution order

To control the execution order, two new decorators need to be added. We will now show a new example with two custom task generator methods 'method1' and 'method2', executed in that order:

[source,python]
---------------
top = '.'
out = 'build'

def configure(conf):
	pass

def build(bld):
	bld(myattr='Hello, world!')

from waflib import TaskGen

@TaskGen.feature('*')
@TaskGen.before('process_source', 'exec_rule')
def method1(self):
	print('method 1 %r' % getattr(self, 'myattr', None))

@TaskGen.feature('*')
@TaskGen.before('process_source')
@TaskGen.after('method1')
def method2(self):
	print('method 2 %r' % getattr(self, 'myattr', None))
---------------

The execution output will be the following:

[source,shishell]
---------------
$ waf distclean configure build --zones=task_gen
'distclean' finished successfully (0.003s)
'configure' finished successfully (0.001s)
Waf: Entering directory `/tmp/simpleproject/build'
15:54:02 task_gen posting objects (normal)
15:54:02 task_gen posting <task_gen of type task_gen defined in dir:///tmp/simpleproject> 139808568487632
15:54:02 task_gen -> method1 (139808568487632)
method 1 'Hello, world!'
15:54:02 task_gen -> exec_rule (139808568487632)
15:54:02 task_gen -> method2 (139808568487632)
method 2 'Hello, world!'
15:54:02 task_gen -> process_source (139808568487632)
15:54:02 task_gen posted
Waf: Leaving directory `/tmp/simpleproject/build'
15:54:02 task_gen posting objects (normal)
'build' finished successfully (0.005s)
---------------

==== Adding or removing a method for execution

The order constraints on the methods (after/before), are used to sort the list of methods in the attribute 'meths'. The sorting is performed once, and the list is consumed as methods are executed. Though no new feature may be added once the first method is executed, new methods may be added dynamically in self.meths. Here is how to create an infinite loop by adding the same method at the end:

[source,python]
---------------
from waflib.TaskGen import feature

@feature('*')
def infinite_loop(self):
	self.meths.append('infinite_loop')
---------------

Likewise, methods may be removed from the list of methods to execute:

[source,python]
---------------
from waflib.TaskGen import feature

@feature('*')
@before_method('process_source')
def remove_process_source(self):
	self.meths.remove('process_source')
---------------

The task generator method workflow is represented in the following illustration:

image::posting{PIC}["Task generator workflow"{backend@docbook:,width=320:},align="center"]

==== Expressing abstract dependencies between task generators

We will now illustrate how task generator methods can be used to express abstract dependencies between task generator objects. Here is a new project file located under '/tmp/targets/':

[source,python]
---------------
top = '.'
out = 'build'

def configure(conf):
	pass

def build(bld):
	bld(rule='echo A', always=True, name='A')
	bld(rule='echo B', always=True, name='B')
---------------

By executing 'waf --targets=B', only the task generator 'B' will create its tasks, and the output will be the following:

[source,shishell]
---------------
$ waf distclean configure build --targets=B
'distclean' finished successfully (0.000s)
'configure' finished successfully (0.042s)
Waf: Entering directory `/tmp/targets/build'
[1/1] B:
B
Waf: Leaving directory `/tmp/targets/build'
'build' finished successfully (0.032s)
---------------

Here is a way to ensure that the task generator 'A' has created its tasks when 'B' does:

[source,python]
---------------
top = '.'
out = 'build'

def configure(conf):
    pass

def build(bld):
    bld(rule='echo A', always=True, name='A')
    bld(rule='echo B', always=True, name='B', depends_on='A')

from waflib.TaskGen import feature, before_method
@feature('*') <1>
@before_method('process_rule')
def post_the_other(self):
    deps = getattr(self, 'depends_on', []) <2>
    for name in self.to_list(deps):
        other = self.bld.get_tgen_by_name(name) <3>
        print('other task generator tasks (before) %s' % other.tasks)
        other.post() <4>
        print('other task generator tasks (after) %s' % other.tasks)
---------------

<1> This method will be executed for all task generators, before the attribute `rule` is processed
<2> Try to process the attribute `depends_on`, if present
<3> Obtain the task generator by name, and for the same variant
<4> Force the other task generator to create its tasks

The output will be:

[source,shishell]
---------------
$ waf distclean configure build --targets=B
'distclean' finished successfully (0.001s)
'configure' finished successfully (0.001s)
Waf: Entering directory `/tmp/targets/build'
other task generator tasks (before) [] <1>
other task generator tasks (after) [ <2>
        {task: A  -> }]
[1/2] B:
B
[2/2] A: <3>
A
Waf: Leaving directory `/tmp/targets/build'
'build' finished successfully (0.014s)
---------------

<1> The other task generator has not created any task yet
<2> A task generator creates all its tasks by calling its method `post()`
<3> Although `--targets=B` was requested, the task from target 'A' was created and executed too

In practice, the dependencies will often re-use the task objects created by the other task generator: node, configuration set, etc. This is used by the uselib system (see the next chapter on c/c++ builds).