waf/docs/book/execution.txt

== Projects and commands

=== Waf commands

==== Waf and wscript files

As the Waf file is meant to be a generic utility for building projects, project-specific details are best kept and versioned in files residing along with the project source code.

The Waf project files are modules written in the Python programming language and are named *wscript*. Though they can contain any Python code, Waf can use specific functions and classes defined in them.

==== Overview of the command-line

When running the following in a terminal or shell, Waf is instructed to run the two commands called *distclean* and *configure* in this specific order. The *-j1* and *--help* elements are command-line options; they are optional and their position or order in the list of arguments is not meant to be significant. The CFLAGS value provides a way to provide arbitrary data in an unverified way, it is also called an environment variable.

[source,shishell]
---------------
$ CFLAGS=-O3 waf distclean configure -j1 --help
---------------

NOTE: waf commands are passed after the 'waf' file and contain no '-' or '=' character

==== Waf commands map Python functions

Waf commands assume that a corresponding command function is defined in a the wscript file in the current folder.
They take a single context parameter as input and do not have to return any particular value as in the following example:

// execution_hello
[source,python]
---------------
#! /usr/bin/env python
# encoding: utf-8

def <1> hello(ctx <2>):
    print('hello world')
---------------

<1> The _waf command_ *hello*
<2> A waf context, used to share data among scripts

Calling the command instructs +waf+ to call function:

[source,shishell]
---------------
$ waf hello
hello world
'hello' finished successfully (0.001s)
---------------

==== Waf command chaining

As previously mentioned, commands are executed in the order defined on the command-line. A wscript file may thus provide an arbitrary amound of commands in the same _wscript_ file:

// execution_ping
[source,python]
---------------
def ping(ctx):
    print(' ping! %d' % id(ctx))

def pong(ctx):
    print(' pong! %d' % id(ctx))
---------------

And such commands may be called more than once by being repeated on the command-line:

[source,shishell]
---------------
$ waf ping pong ping ping
 ping! 140704847272272
'ping' finished successfully (0.001s)
 pong! 140704847271376
'pong' finished successfully (0.001s)
 ping! 140704847272336
'ping' finished successfully (0.001s)
 ping! 140704847272528
'ping' finished successfully (0.001s)
---------------

NOTE: Command functions are passed a new context object when they are called; the class for that object is command-specific: ConfigureContext for configure, BuildContext for build, OptionContext for option, and Context for any other command.

==== Using several scripts and folders

Although a Waf project must contain a top-level _wscript_ file, the contents may be split into several sub-project files. We will now illustrate this concept on a small project:

[source,shishell]
---------------
$ tree
|-- src
|   `-- wscript
`-- wscript
---------------

The commands in the top-level _wscript_ will call the same commands from a subproject _wscript_ file by calling a context method named _recurse_:

// execution_recurse
[source,python]
---------------
def ping(ctx):
	print('→ ping from ' + ctx.path.abspath())
	ctx.recurse('src')
---------------

And here is the contents of 'src/wscript'

[source,python]
---------------
def ping(ctx):
	print('→ ping from ' + ctx.path.abspath())
---------------

Upon execution, the results will be:

[source,shishell]
---------------
$ cd /tmp/execution_recurse

$ waf ping
→ ping from /tmp/execution_recurse
→ ping from /tmp/execution_recurse/src
'ping' finished successfully (0.002s)

$ cd src

$ waf ping
→ ping from /tmp/execution_recurse/src
'ping' finished successfully (0.001s)
---------------

NOTE: The method _recurse_, and the attribute _path_ are available on all waf context classes

=== Waf project definition

==== Configuring a project (the _configure_ command)

Though Waf may be called from any folder containing a 'wscript' file, an entry point must be defined in a particular project file.
This lifts ambiguities and saves the redefinition of the same imports and function definitions in all sub-wscript files of a project.
The following concepts help to structure a Waf project:

. Project directory: directory containing the source files that will be packaged and redistributed to other developers or to end users
. Build directory: directory containing the files generated by the project (configuration sets, build files, logs, etc)
. System files: files and folders which do not belong to the project (operating system files, etc)

The predefined command named _configure_ is used to gather and store the information about these folders.
We will now extend the example from the previous section with the following top-level wscript file:

// execution_configure
[source,python]
---------------
top = '.' <1>
out = 'build_directory' <2>

def configure(ctx): <3>
	print('→ configuring the project in ' + ctx.path.abspath())

def ping(ctx):
	print('→ ping from ' + ctx.path.abspath())
	ctx.recurse('src')
---------------

<1> string representing the project directory. In general, top is set to '.', except for some proprietary projects where the wscript cannot be added to the top-level, top may be set to '../..' or even some other folder such as '/checkout/perforce/project'
<2> string representing the build directory. In general, it is set to 'build', except for some proprietary projects where the build directory may be set to an absolute path such as '/tmp/build'. It is important to be able to remove the build directory safely, so it should never be given as '.' or '..'.
<3> the _configure_ function is called by the 'configure' command

The script in 'src/wscript' is left unchanged:

[source,python]
---------------
def ping(ctx):
	print('→ ping from ' + ctx.path.abspath())
---------------

The execution output will be the following:

////
$ waf ping
→ ping from /tmp/execution_configure
→ ping from /tmp/execution_configure/src
'ping' finished successfully (0.001s)

$ cd src

$ waf ping
→ ping from /tmp/execution_configure/src
'ping' finished successfully (0.001s)

$ cd ..

////

[source,shishell]
---------------
$ cd /tmp/execution_configure <1>
$ tree
|-- src
|   `-- wscript
`-- wscript

$ waf configure <2>
→ configuring the project in /tmp/execution_configure
'configure' finished successfully (0.021s)

$ tree -a
|-- build_directory/ <3>
|   |-- c4che/ <4>
|   |   |-- build.config.py <5>
|   |   `-- _cache.py <6>
|   `-- config.log <7>
|--.lock-wafbuild <8>
|-- src
|   `-- wscript
`-- wscript

$ waf ping
→ ping from /tmp/execution_configure
→ ping from /tmp/execution_configure/src
'ping' finished successfully (0.001s)

$ cd src
$ waf ping <9>
→ ping from /tmp/execution_configure
→ ping from /tmp/execution_configure/src
'ping' finished successfully (0.001s)
---------------

<1> To configure the project, change to the directory containing the top-level project file
<2> The execution is called by calling _waf configure_
<3> The build directory was created
<4> The configuration data is stored in the folder 'c4che/'
<5> The command-line options and environment variables in use are stored in 'build.config.py'
<6> The user configuration set is stored in '_cache.py'
<7> Configuration log (duplicate of the output generated during the configuration)
<8> Hidden file pointing at the relevant project file and build directory
<9> Calling _waf_ from a subfolder will execute the commands from the same wscript file used for the configuration

NOTE: _waf configure_ is always called from the directory containing the wscript file

==== Removing generated files (the _distclean_ command)

A command named _distclean_ is provided to remove the build directory and the lock file created during the configuration. On the example from the previous section:

[source,shishell]
---------------
$ waf configure
→ configuring the project in /tmp/execution_configure
'configure' finished successfully (0.001s)

$ tree -a
|-- build_directory/
|   |-- c4che/
|   |   |-- build.config.py
|   |   `-- _cache.py
|   `-- config.log
|--.lock-wafbuild
`-- wscript

$ waf distclean <1>
'distclean' finished successfully (0.001s)

$ tree <2>
|-- src
|   `-- wscript
`-- wscript
---------------

<1> The _distclean_ command definition is implicit (no declaration in the wscript file)
<2> The tree is reverted to its original state: no build directory and no lock file

The behaviour of _distclean_ is fairly generic and the corresponding function does not have to be defined in the wscript files. It may be defined to alter its behaviour though, see for example the following:

[source,python]
---------------
top = '.'
out = 'build_directory'

def configure(ctx):
	print('→ configuring the project')

def distclean(ctx):
	print(' Not cleaning anything!')
---------------

Upon execution:

[source,shishell]
---------------
$ waf distclean
 Not cleaning anything!
'distclean' finished successfully (0.000s)
---------------

==== Packaging the project sources (the _dist_ command)

The _dist_ command is provided to create an archive of the project. By using the script presented previously:

// execution_dist

[source,python]
---------------
top = '.'
out = 'build_directory'

def configure(ctx):
	print('→ configuring the project in ' + ctx.path.abspath())
---------------

Execute the _dist_ command to get:

[source,shishell]
---------------
$ cd /tmp/execution_dist

$ waf configure
→ configuring the project in /tmp/execution_dist
'configure' finished successfully (0.005s)

$ waf dist
New archive created: noname-1.0.tar.bz2 (sha='a4543bb438456b56d6c89a6695f17e6cb69061f5')
'dist' finished successfully (0.035s)
---------------

By default, the project name and version are set to 'noname' and '1.0'. To change them, it is necessary to provide two additional variables in the top-level project file:

[source,python]
---------------
APPNAME = 'webe'
VERSION = '2.0'

top = '.'
out = 'build_directory'

def configure(ctx):
	print('→ configuring the project in ' + ctx.path.abspath())
---------------

Because the project was configured once, it is not necessary to configure it once again:

[source,shishell]
---------------
$ waf dist
New archive created: webe-2.0.tar.bz2 (sha='7ccc338e2ff99b46d97e5301793824e5941dd2be')
'dist' finished successfully (0.006s)
---------------

More parameters may be given to alter the archive by adding a function 'dist' in the script:

[source,python]
---------------
def dist(ctx):
	ctx.base_name = 'foo_2.0' <1>
	ctx.algo      = 'zip' <2>
	ctx.excl      = ' **/.waf-1* **/*~ **/*.pyc **/*.swp **/.lock-w*' <3>
	ctx.files     = ctx.path.ant_glob('**/wscript') <4>
---------------

<1> The archive name may be given directly instead of computing from 'APPNAME' and 'VERSION'
<2> The default compression format is 'tar.bz2'. Other valid formats are 'zip' and 'tar.gz'
<3> Exclude patterns passed to give to 'ctx.path.ant_glob()' which is used to find the files
<4> The files to add to the archive may be given as Waf node objects ('excl' is therefore ignored)

==== Defining command-line options (the _options_ command)

The Waf script provides various default command-line options, which may be consulted by executing +waf --help+:

[source,shishell]
---------------
$ waf --help
waf [command] [options]

Main commands (example: ./waf build -j4)
  build    : executes the build
  clean    : cleans the project
  configure: configures the project
  dist     : makes a tarball for redistributing the sources
  distcheck: checks if the project compiles (tarball from 'dist')
  distclean: removes the build directory
  install  : installs the targets on the system
  list     : lists the targets to execute
  step     : executes tasks in a step-by-step fashion, for debugging
  uninstall: removes the targets installed

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -j JOBS, --jobs=JOBS  amount of parallel jobs (2)
  -k, --keep            keep running happily even if errors are found
  -v, --verbose         verbosity level -v -vv or -vvv [default: 0]
  --zones=ZONES         debugging zones (task_gen, deps, tasks, etc)

  configure options:
    -o OUT, --out=OUT   build dir for the project
    -t TOP, --top=TOP   src dir for the project
    --prefix=PREFIX     installation prefix [default: '/usr/local/']
    --download          try to download the tools if missing

  build and install options:
    -p, --progress      -p: progress bar; -pp: ide output
    --targets=TARGETS   task generators, e.g. "target1,target2"

  step options:
    --files=FILES       files to process, by regexp, e.g. "*/main.c,*/test/main.o"

  install/uninstall options:
    --destdir=DESTDIR   installation root [default: '']
    -f, --force         force file installation
---------------

Accessing a command-line option is possible from any command. Here is how to access the value _prefix_:

[source,python]
---------------
top = '.'
out = 'build_directory'

def configure(ctx):
	print('→ prefix is ' + ctx.options.prefix)
---------------

Upon execution, the following will be observed:

[source,shishell]
---------------
$ waf configure
→ prefix is /usr/local/
'configure' finished successfully (0.001s)
---------------

To define project command-line options, a special command named _options_ may be defined in user scripts. This command will be called once before any other command executes.

[source,python]
---------------
top = '.'
out = 'build_directory'

def options(ctx):
	ctx.add_option('--foo', action='store', default=False, help='Silly test')

def configure(ctx):
	print('→ the value of foo is %r' % ctx.options.foo)
---------------

Upon execution, the following will be observed:

[source,shishell]
---------------
$ waf configure --foo=test
→ the value of foo is 'test'
'configure' finished successfully (0.001s)
---------------

The command context for options is a shortcut to access the optparse functionality. For more information on the optparse module, consult the http://docs.python.org/library/optparse.html[Python documentation]


=== The _build_ commands

==== Building targets (the _build_ command)

The 'build' command is used for building targets. We will now create a new project in '/tmp/execution_build/', and add a script to create an empty file +foo.txt+ and then copy it into another file +bar.txt+:

// execution_build
[source,python]
---------------
top = '.'
out = 'build_directory'

def configure(ctx):
    pass

def build(ctx):
    ctx(rule='touch ${TGT}', target='foo.txt')
    ctx(rule='cp ${SRC} ${TGT}', source='foo.txt', target='bar.txt')
---------------

Calling _waf build_ directly results in an error:

[source,shishell]
---------------
$ cd /tmp/execution_build/

$ waf build
The project was not configured: run "waf configure" first!
---------------

The build requires a configured folder to know where to look for source files and where to output the created files. Let's try again:

[source,shishell]
---------------
$ waf configure build
'configure' finished successfully (0.007s)
Waf: Entering directory `/tmp/execution_build/build_directory'
[1/2] foo.txt:  -> build_directory/foo.txt <1>
[2/2] bar.txt: build_directory/foo.txt -> build_directory/bar.txt
Waf: Leaving directory `/tmp/examples/execution_build/build_directory'
'build' finished successfully (0.041s)

$ tree -a
|-- build_directory/
|   |-- bar.txt <2>
|   |-- c4che/
|   |   |-- build.config.py
|   |   `-- _cache.py
|   |-- foo.txt
|   |-- config.log
|   `-- .wafpickle <3>
|--.lock-wafbuild
`-- wscript

$ waf build
Waf: Entering directory `/tmp/execution_build/build_directory'
Waf: Leaving directory `/tmp/execution_build/build_directory'
'build' finished successfully (0.008s) <4>
---------------

<1> Note that the build _deduced_ that +bar.txt+ has to be created after +foo.txt+
<2> The targets are created in the build directory
<3> A pickle file is used to store the information about the targets
<4> Since the targets are up-to-date, they do not have to be created once again

Since the command _waf build_ is usually executed very often, a shortcut is provided to call it implicitly:

[source,shishell]
---------------
$ waf
Waf: Entering directory `/tmp/execution_build/build_directory'
Waf: Leaving directory `/tmp/execution_build/build_directory'
---------------

==== Cleaning targets (the _clean_ command)

The _clean_ command is used to remove the information about the files and targets created during the build. It uses the same function _build_ from the wscript files so there is no need to add a function named _clean_ in the wscript file.

After cleaning, the targets will be created once again even if they were up-to-date:

[source,shishell]
---------------
$ waf clean build -v
'clean' finished successfully (0.003s)
Waf: Entering directory `/tmp/execution_build/build_directory' <1>
[1/2] foo.txt:  -> build_directory/foo.txt <2>
14:58:34 runner 'touch foo.txt' <3>
[2/2] bar.txt: build_directory/foo.txt -> build_directory/bar.txt
14:58:34 runner 'cp foo.txt bar.txt'
Waf: Leaving directory `/tmp/execution_build/build_directory'
'build' finished successfully (0.040s)
---------------

<1> All commands are executed from the build directory by default
<2> The information about the files +foo.txt+ was lost so it is rebuilt
<3> By using the _-v_ flag, the command-lines executed are displayed

==== More build commands

The following commands all use the same function _build_ from the wscript file:

. +build:+ process the source code to create the object files
. +clean:+ remove the object files that were created during a build (unlike distclean, do not remove the configuration)
. +install:+ check that all object files have been generated and copy them on the system (programs, libraries, data files, etc)
. +uninstall:+ undo the installation, remove the object files from the system without touching the ones in the build directory
. +list:+ list the task generators in the build section (to use with waf --targets=name)
. +step:+ force the rebuild of particular files for debugging purposes

The attribute 'cmd' holds the name of the command being executed:

// execution_cmd
[source,python]
---------------
top = '.'
out = 'build_directory'

def configure(ctx):
	print(ctx.cmd)

def build(ctx):
	if ctx.cmd == 'clean':
		print('cleaning!')
	else:
		print(ctx.cmd)
---------------

The execution will produce the following output:

[source,shishell]
---------------
$ waf configure clean build
Setting top to : /tmp/execution_cmd
Setting out to : /tmp/execution_cmd/build_directory
configure
'configure' finished successfully (0.002s)
cleaning!
'clean' finished successfully (0.002s)
Waf: Entering directory `/tmp/execution_cmd/build_directory'
build
Waf: Leaving directory `/tmp/execution_cmd/build_directory'
'build' finished successfully (0.001s)
---------------

The build command usage will be described in details in the next chapters.