waf/docs/book/architecture.txt

== Waf architecture overview

This chapter provides describes the Waf library and the interaction between the components.

=== Modules and classes

==== Core modules

Waf consists of the following modules which constitute the core library. They are located in the directory `waflib/`. The modules located under `waflib/Tools` and `waflib/extras` are extensions which are not part of the Waf core.

.List of core modules
[options="header", cols="1,6"]
|=================
|Module    | Role
|Build     | Defines the build context classes (build, clean, install, uninstall), which holds the data for one build (paths, configuration data)
|Configure | Contains the configuration context class, which is used for launching configuration tests and writing the configuration settings for the build
|ConfigSet | Contains a dictionary class which supports a lightweight copy scheme and provides persistence services
|Context   | Contains the base class for all waf commands (context parameters of the Waf commands)
|Errors    | Exceptions used in the Waf code
|Logs      | Loggging system wrapping the calls to the python logging module
|Node      | Contains the file system representation class
|Options   | Provides a custom command-line option processing system based on optparse
|Runner    | Contains the task execution system (thread-based producer-consumer)
|Scripting | Constitutes the entry point of the Waf application, executes the user commands such as build, configuration and installation
|TaskGen   | Provides the task generator system, and its extension system based on method addition
|Task      | Contains the task class definitions, and factory functions for creating new task classes
|Utils     | Contains support functions and classes used by other Waf modules
|=================

Not all core modules are required for using Waf as a library. The dependencies between the modules are represented on the following diagram. For example, the module 'Node' requires both modules 'Utils' and 'Errors'. Conversely, if the module 'Build' is used alone, then the modules 'Scripting' and 'Configure' can be removed safely.

image::core{PIC}["Module dependencies"{backend@docbook:,height=400:},align="center"]

==== Context classes

User commands, such as 'configure' or 'build', are represented by classes derived from 'waflib.Context.Context'. When a command does not have a class associated, the base class 'waflib.Context.Context' is used instead.

The method 'execute' is the start point for a context execution, it often calls the method 'recurse' to start reading the user scripts and execute the functions referenced by the 'fun' class attribute.

The command is associated to a context class by the class attribute 'cmd' set on the class. Context subclasses are added in 'waflib.Context.classes' by the metaclass 'store_context' and loaded through the function 'waflib.Context.create_context'. The classes defined last will replace existing commands.

As an example, the following context class will define or override the 'configure' command. When calling 'waf configure', the function 'foo' will be called from wscript files:

[source,python]
---------------
from waflib.Context import Context
class somename(Context):
    cmd = 'configure'
    fun = 'foo'
---------------

image::classes{PIC}["Context classes"{backend@docbook:,width=500:},align="center"]

==== Build classes

The class 'waflib.Build.BuildContext' and its subclasses such as 'waflib.Build.InstallContext' or 'waflib.Build.StepContext' have task generators created when reading the user scripts. The task generators will usually have task instances, depending on the operations performed after all task generators have been processed.

The 'ConfigSet' instances are copied from the build context to the tasks ('waflib.ConfigSet.ConfigSet.derive') to propagate values such as configuration flags. A copy-on-write is performed through most methods of that class (append_value, prepend_value, append_unique).

The 'Parallel' object encapsulates the iteration over all tasks of the build context, and delegates the execution to thread objects (producer-consumer).

The overall structure is represented on the following diagram:

image::classes_build{PIC}["Build classes"{backend@docbook:,width=500:},align="center"]

=== Context objects

==== Context commands and recursion

The context commands are designed to be as independent as possible, and may be executed concurrently. The main application is the execution of small builds as part of configuration tests. For example, the method 'waflib.Tools.c_config.run_c_code' creates a private build context internally to perform the tests.
Here is an example of a build that creates and executes simple configuration contexts concurrently:

// architecture_link
[source,python]
---------------
import os
from waflib.Configure import conf, ConfigurationContext
from waflib import Task, Build, Logs

def options(ctx):
	ctx.load('compiler_c')

def configure(ctx):
	ctx.load('compiler_c')

def build(ctx):
	ctx(rule=run_test, always=True, header_name='stdio.h') <1>
	ctx(rule=run_test, always=True, header_name='unistd.h')

def run_test(self):
	top = self.generator.bld.srcnode.abspath()
	out = self.generator.bld.bldnode.abspath()

	ctx = ConfigurationContext(top_dir=top, out_dir=out) <2>
	ctx.init_dirs() <3>

	ctx.in_msg = 1 <4>
	ctx.msg('test') <5>

	header = self.generator.header_name
	logfile = self.generator.path.get_bld().abspath() + os.sep \
		+ header + '.log'
	ctx.logger = Logs.make_logger(logfile, header) <6>

	ctx.env = self.env.derive() <7>
	ctx.check(header_name=header) <8>
---------------

<1> Create task generators which will run the method 'run_test' method defined below
<2> Create a new configuration context as part of a 'Task.run' call
<3> Initialize ctx.srcnode and ctx.bldnode (build and configuration contexts only)
<4> Set the internal counter for the context methods 'msg', 'start_msg' and 'end_msg'
<5> The console output is disabled (non-zero counter value to disable nested messages)
<6> Each context may have a logger to redirect the error messages
<7> Initialize the default environment to a copy of the task one
<8> Perform a configuration check

After executing 'waf build', the project folder will contain the new log files:

[source,shishell]
---------------
$ tree
.
|-- build
|   |-- c4che
|   |   |-- build.config.py
|   |   `-- _cache.py
|   |-- config.log
|   |-- stdio.h.log
|   `-- unistd.h.log
`-- wscript
---------------

A few measures are set to ensure that the contexts can be executed concurrently:

. Context objects may use different loggers derived from the 'waflib.Logs' module.
. Each context object is associated to a private subclass of 'waflib.Node.Node' to ensure that the node objects are unique. To pickle Node objects, it is important to prevent concurrent access by using the lock object 'waflib.Node.pickle_lock'.

==== Build context and persistence

The build context holds all the information necessary for a build. To accelerate the start-up, a part of the information is stored and loaded between the runs. The persistent attributes are the following:

.Persistent attributes
[options="header", cols="1,3,3"]
|=================
|Attribute | Description                                            | Type
|root      | Node representing the root of the file system          | Node
|node_deps | Implicit dependencies                                  | dict mapping Node to signatures
|raw_deps  | Implicit file dependencies which could not be resolved | dict mapping Node ids to any serializable type
|task_sigs | Signature of the tasks executed                        | dict mapping a Task computed uid to a hash
|=================


=== Support for c-like languages

==== Compiled tasks and link tasks

The tool _waflib.Tools.ccroot_ provides a system for creating object files and linking them into a single final file. The method _waflib.Tools.ccroot.apply_link_ is called after the method _waflib.TaskGen.process_source_ to create the link task. In pseudocode:

[source,shishell]
---------------
call the method process_source:
  for each source file foo.ext:
    process the file by extension
      if the method create_compiled_task is used:
        create a new task
        set the output file name to be foo.ext.o
        add the task to the list self.compiled_tasks

call the method apply_link
  for each name N in self.features:
    find a class named N:
      if the class N derives from 'waflib.Tools.ccroot.link_task':
        create a task of that class, assign it to self.link_task
        set the link_task inputs from self.compiled_tasks
        set the link_task output name to be env.N_PATTERN % self.target
        stop
---------------

This system is used for _assembly_, _C_, _C++_, _D_ and _fortran_ by default. Note that the method _apply_link_ is supposed to be called after the method _process_source_.

We will now demonstrate how to support the following mini language:

[source,shishell]
---------------
cp: .ext -> .o
cat: *.o -> .exe
---------------

Here is the project file:

// architecture_link
[source,python]
---------------
def configure(ctx):
	pass

def build(ctx):
	ctx(features='mylink', source='foo.ext faa.ext', target='bingo')

from waflib.Task import Task
from waflib.TaskGen import feature, extension, after_method
from waflib.Tools import ccroot <1>

@after_method('process_source')
@feature('mylink')
def call_apply_link(self): <2>
	self.apply_link()

class mylink(ccroot.link_task): <3>
	run_str = 'cat ${SRC} > ${TGT}'

class ext2o(Task):
	run_str = 'cp ${SRC} ${TGT}'

@extension('.ext')
def process_ext(self, node):
	self.create_compiled_task('ext2o', node) <4>
---------------

<1> This import will bind the methods such as _create_compiled_task_ and _apply_link_task_
<2> An alternate definition would be calling _waflib.TaskGen.feats[`mylink'] = [`apply_link']_
<3> The link task must be a subclass of another link task class
<4> Calling the method _create_compiled_task_

The execution outputs will be the following:
// why the extra space after "setting top to"?
[source,shishell]
---------------
$ waf distclean configure build -v
'distclean' finished successfully (0.005s)
Setting top to   : /tmp/architecture_link
Setting out to   : /tmp/architecture_link/build
'configure' finished successfully (0.008s)
Waf: Entering directory `/tmp/architecture_link/build'
[1/3] ext2o: foo.ext -> build/foo.ext.0.o
12:50:25 runner ['cp', '../foo.ext', 'foo.ext.0.o']
[2/3] ext2o: faa.ext -> build/faa.ext.0.o
12:50:25 runner ['cp', '../faa.ext', 'faa.ext.0.o']
[3/3] mylink: build/foo.ext.0.o build/faa.ext.0.o -> build/bingo
12:50:25 runner 'cat foo.ext.0.o faa.ext.0.o > bingo'
Waf: Leaving directory `/tmp/architecture_link/build'
'build' finished successfully (0.041s)
---------------

NOTE: Task generator instances have at most one link task instance




=== Writing re-usable Waf tools

==== Adding a waf tool

===== Importing the code

The intent of the Waf tools is to promote high cohesion by moving all conceptually related methods and classes into separate files, hidden from the Waf core, and as independent from each other as possible.

Custom Waf tools can be left in the projects, added to a custom waf file through the 'waflib/extras' folder, or used through 'sys.path' changes.

The tools can import other tools directly through the 'import' keyword. The scripts however should always import the tools to the 'ctx.load' to limit the coupling. Compare for example:

[source,python]
---------------
def configure(ctx):
    from waflib.extras.foo import method1
    method1(ctx)
---------------

and:

[source,python]
---------------
def configure(ctx):
    ctx.load('foo')
    ctx.method1()
---------------

The second version should be preferred, as it makes fewer assumptions on whether 'method1' comes from the module 'foo' or not, and on where the module 'foo' is located.

===== Naming convention for C/C++/Fortran

The tools 'compiler_c', 'compiler_cxx' and 'compiler_fc' use other waf tools to detect the presense of particular compilers. They provide a particular naming convention to give a chance to new tools to register themselves automatically and save the import in user scripts. The tools having names beginning by 'c_', 'cxx_' and 'fc_' will be tested.

The registration code will be similar to the following:

[source,python]
---------------
from waflib.Tools.compiler_X import X_compiler
X_compiler['platform'].append('module_name')
---------------

where *X* represents the type of compiler ('c', 'cxx' or 'fc'), *platform* is the platform on which the detection should take place (linux, win32, etc), and *module_name* is the name of the tool to use.

==== Command methods

===== Subclassing is only for commands

As a general rule, subclasses of 'waflib.Context.Context' are created only when a new user command is necessary. This is the case for example when a command for a specific variant (output folder) is required, or to provide a new behaviour. When this happens, the class methods 'recurse', 'execute' or the class attributes 'cmd', 'fun' are usually overridden.

NOTE: If there is no new command needed, do not use subclassing.

===== Domain-specific methods are convenient for the end users

Although the Waf framework promotes the most flexible way of declaring tasks through task generators, it is often more convenient to declare domain-specific wrappers in large projects. For example, the samba project provides a function used as:

[source,python]
---------------
bld.SAMBA_SUBSYSTEM('NDR_NBT_BUF',
    source    = 'nbtname.c',
    deps      = 'talloc',
    autoproto = 'nbtname.h'
    )
---------------

===== How to bind new methods

New methods are commonly bound to the build context or to the configuration context by using the '@conf' decorator:

[source,python]
---------------
from waflib.Configure import conf

@conf
def enterprise_program(self, *k, **kw):
    kw['features'] = 'c cprogram debug_tasks'
    return self(*k, **kw)

def build(bld):
    # no feature line
    bld.enterprise_program(source='main.c', target='app')
---------------

The methods should always be bound in this manner or manually, as subclassing may create conflicts between tools written for different purposes.