As a regression / misbehaviour compared to waf 2.0 when a Python
module is installed it seems that, even tho `PREFIX` is set for the whole
project, it is ignored.
In the past this was not ignored, namely:
```
(pydir,) = conf.get_python_variables(["get_python_lib(standard_lib=0, prefix=%r) or ''" % conf.env.PREFIX])
```
While with the current version we would always get (if not forced
with specific `PYTHONDIR` and `PYTHONARCHDIR`) the hardcoded path defined
at compilation time (`/usr/local/lib` in all cases I saw)
This patch takes this into account, using the user schema that permits to
customize the output passing the userbase variable, which is set to `PREFIX`
value indeed.
sysconfig.get_preferred_scheme('user') could give us the user schema, but
this is Python >= 3.10 only, so instead we construct the name manually
depending on the OS.
- When waf is run with -v, and it runs a call to context.cmd_and_log() with an argument list,
argument[0] is a relative path, and a cwd **kwarg is passed so that the argument[0] resolves
correctly, then the call will crash saying the program could not be found. For example, the
caller may be wrapping calls using a nodejs environment like:
ctx.cmd_and_log("./node_modules/bin/webpack", cwd="webui")
and this will fail with "Program ./node_modules/.bin/webpack not found!"
if waf is run with -v. The user friendly check for usable programs still
stays in place for shell calls and absolute paths, but allows the caller
to use this pattern even when verbose mode is on. This same fix was
previously made for context.exec_command().
- When waf is run with -v, and it runs a call to context.exec_command() with an argument list,
argument[0] is a relative path, and a cwd **kwarg is passed so that the argument[0] resolves
correctly, then the call will crash saying the program could not be found. For example, the
caller may be wrapping calls using a nodejs environment like:
ctx.exec_command("./node_modules/bin/webpack", cwd="webui")
and this will fail with "Program ./node_modules/.bin/webpack not found!"
if waf is run with -v. The user friendly check for usable programs still
stays in place for shell calls and absolute paths, but allows the caller
to use this pattern even when verbose mode is on.
Implements support for Qt6 by extending qt5.py. The user can opt in for
Qt6 support by setting cfg.want_qt6 = True. There's also a qt6 feature,
which at the moment is identical to the qt5 feature. Splitting has been
done now for futureproofing purposes. Qt6 libraries can be selected
through the cfg.qt6_vars variable. I didn't make an attempt at any
backwards compatibility by trying to load cfg.qt5_vars if it exists,
this is done so the move from Qt5 to Qt6 is a more deliberate process.
Signed-off-by: Rafaël Kooi <3961583-RA-Kooi@users.noreply.gitlab.com>
Improves autodetection by adding tool naming as found in some recent
distributions (ie. Fedora). Adds also possibility to pass via env
command line options to tools (needed ie. to explicitly pass
generator due to changes to uic/rcc tool).
Test updated to demonstrate and document the parameter needed to
work out of the box with newest tooling.
shlib is no-op in some baremetal newlib based toolchains (for example in
riscv one), and causes the check to fail as the --dynamic flag is not
recognized
Make it easy to add custom target executions in the automatic
eclipse configuration generation, for example to call other
standard waf targets from other tools or with specific options.
Make sure just unique include paths (both system and local) are
added to prevent overcrowding with useless redundant include paths
that grow up a lot the generated XML file and make the usage of
the GUI messy.
The filter was already there for Java/Python.
Add automatic generation of editor language settings for C and C++,
so the automatic code correction uses the correct compiler and
compiler flags, including for example the correct C/C++ standard
so construct from such standards are correctly managed by the IDE.
Correct compiler and flags are automatically generated using the
build environment data gathered during configure phase.
The playground example has been modified to contain some code that
is standard specific to demonstrate the new feature when run under
Eclipse.
In the current implementation if a project is using
build variants it's not possible to use the clang_compilation_database
plugin because it strips the variant information from the build object.
Rework how gccdeps' cached_nodes lock is used so acquiring the lock is
only necessary on a cache miss. Also use a "with" context manager to
simplify management of the lock lifecycle.
Ported from 8b5a2a2086
Move the scan() method down in the file to match msvcdeps' method
ordering. This makes it easier to compare gccdeps.py and msvcdeps.py
to keep them in sync.
Visual Studio returns paths to dependencies with incorrect case.
ant_glob() is very slow for this use case (40~50% impact to overall
build time). This patch uses os.listdir() to find the correct case
of each path component.
In order to correctly set a default project in visual studio any folders
must be listed at the top of the solution file. This change ensures that
any folders included in generated solutions sort to the top of the .sln
file. The default project, if one exists, will be located after the
folders. Note that it should also be correct to place the default
at the top of the file, followed by any folders.
The unit test tool moved from a simple split to using shlex.split for
handling the unit test command. This results in the path separators on
windows being treated as escapes.
To handle this the unit test exec command is properly escaped before
joining so that the subsequent split restores the original arguments.
The quote function is also exposed in the Utilities module so that
wscripts making use of the unit test tool can properly quote their
contributions to the command as well.
Haxe support
This commit adds support for haxe over [lix](https://github.com/lix-pm/lix.client) toolkit
- haxe library validation: check and fetch missing libs if needed
- "haxe" loader with "hx" compiler
- HAXEFLAGS
- lib checking and uselib_store support
- ctx.haxe with `res` argument to be more simple
- error checking
See merge request ita1024/waf!2308
- Exclude classes having folder or symlinks
- Exclude well-known Task classes from wafcache processing
- Remove stale 'waflib.Task.Task.chmod' processing
Add support for MinIO object storage (https://min.io/) using the
MinIO client (https://github.com/minio/mc) to wafcache.
MinIO is an open-source, self-hostable, S3 compatible cache. The
MinIO client supports MinIO connections as well as normal S3/GCS
storages by configuring aliases beforehand.
Hint: some distributions have `mc` (the GNU Midnight Commander)
installed which is not the minio client, be aware of this (or your
build may get stuck with waf waiting for `mc` to never finish)
Print out which source file waf is gathering dependencies for and leave
the leading spaces in the dependency debug output because it can be
helpful to see the dependency hierarchy.
Add support for generating and using gcc's native dependency files with
the GNU Assembler in addition to the existing C/C++ support.
When the gas and gccdeps tools are loaded, the configure step will test
whether gcc operating on an assembly file supports the -MMD argument.
If so, waf will pass the -MMD argument to .S files assembled with gcc
which will cause it to generate .d dependency files. Waf will then parse
those files for dependency information.
Note: This will only work for assembly files compiled through the gcc
frontend, not with GNU as directly. It also requires assembly files to
use the uppercase .S file extension.
variable x is used in the outer loop and gets corrupted by inner enumeration in case of non-jar dependency
to reproduce: use the demos/java and run waf build twice: the first time will work (since no class files around)
while the second will not since will by bad luck pick a class file in the inner loop
clang_compilation_database: fix#2247, add clangdb command to generate database by request without rebuilding, add tests (WIP)
Closes#2247
See merge request ita1024/waf!2256
When using msvcdeps, header dependencies are not detected reliably for
generated source files. The root cause is a bug in versions of MSVC
prior to VS2019 16.0 in which it emits lower-case path prefixes when
resolving include paths relative to the containing file. Absolute paths
and paths relative to include directories passed in the MSVC command
line are, in contrast, case-correct.
Such a file-relative include directive with an incorrect lower-case
prefix derails waf's node hash signature handling and fails silently.
This change uses ant_glob() with the ignorecase keyword argument to
find the file on the filesystem with the correct case. The prior
case-correction code has been superseded and was removed.
See the following Visual Studio bug report for details on the issue:
https://developercommunity.visualstudio.com/content/problem/233871/showincludes-lowercases-some-path-segments.html
Make path_to_node() only accept a path as a string instead of also as a
list. That requires joining the list of path components in the relative
path case before calling path_to_node(). Also use path.pop(0) to remove
the first path component instead of copying the remainder of the path
using a slice operator.
Rework how msvcdeps' cached_nodes lock is used so acquiring the lock is
only necessary on a cache miss. Also use a "with" context manager to
simplify management of the lock lifecycle.
ant_matcher() converts an ANT glob pattern to an equivalent regex
pattern. This commit adds support for escaping parenthesis in the
input pattern so they don't end up being treated as a regex capture
group.
Also add a unit test to verify ant_glob()'s ability to handle special
characters in the input pattern.
Previously one could explicitly state to use PySide2 or PyQt4 but not PyQt5 which was picked just by default. In this way the option can override local configurations and also this prevents to have mixed tools versions if we are sure we need PyQt5.
This patch corrects an error in the exec_response_command exception
handler which always assumed that the execution's stdout would be bound
to the the WafError exception object.
However, this assumption is only true when the execution completes with
a non-zero status code. For other exceptions, the stdout attribute is
not bound.
Now, when stdout is not available, the WafError msg will be used
instead.
The order of the lines in a doxyfile are important. This patch uses an
ordered dictionary to keep the keys of the doxyfile in the same order.
This is particularly important for doxyfiles that contain @INCLUDE
lines. In such cases, if the dictionary is not ordered, the @INCLUDE
line can end up in the middle of the generated doxyfile and thus
override all entries that were seen before it.
Currently PDBs are only installed if the /DEBUG flag appears in the
current toolchain's LINKFLAGS attribute. This patch expands support
so that /DEBUG:FULL and /DEBUG:FASTLINK also cause PDBs to be
installed.
mac-o symbols are prefixed with an underscore. when specifying multiple
sub-regexes (e.g. 'sym1|sym2|sym3'), only the first will be matched
(since the expansion turns into '(?P<symbol>_?sym1|sym2|sym3)'). here,
this is remedied by wrapping the symbol regex in a paren group.
This patch attempts to detects if, when we are running from within an
MSYS2 environement (MSYSTEM is set) we are also executing inside an
MSYS2 provided version of python. It does this by assuming that if we
are not in a cygwin environment and we are building on windows, If the
value of sys.executable is /usr/bin or /bin or /usr/local/bin (somethign
unixy) then we are running in an MSYS2 python interpreter and shoudl
compensate for msys2 root paths. Otherwise we shouldn't be doing extra
path manipulation.
Not all tools executed by tasks support the '@argsfile' syntax for
shunting commandline arguments to a file. This means that if such
commands are shunted to a file early, he command will not work. On
windows the rc.exe command is such an example, but some tools on linux
have similar limitations. In the posix case, we artifically limit our
commandline size because it is difficult/variable to caluclate what the
actual limit is (it is partially dependent on environment size). This
could artifically cause commands to fail due to commandline length when
they otherwise wouldn't.
This patch fixes this issue by adding the 'allow_argsfile' flag to the
task. This way certain task instances will be able to specify if they
are compatible with the '@argsfile' syntax or not.
This patch addresses the bug described in issue #2225 where in using
posix paths and an empty PREFIX value can result in files being
installed to the root of the drive specified by destdir instead of to
the desired prefix value. This is a bug in the assumption that user
specified paths that are strings will contain directory separators that
match the target operating system.
The previous patches to workaround
http://support.microsoft.com/kb/830473 drastically over estimated the
number of characters in commands by treating the repr() version of the
command array as a reasonable estimator of commandline length. This
caused commands attempt to write argsfiles before they should have.
The new calculation calculates the number characters in the command
array and adds the number of spaces that would be added by ' '.join(cmd)
this provides a much closer estimate of the commandline length.
This also limits the CLI-length on non windows platforms to 200kB. This
prevents us hitting the much larger argument limits on Linux/BSD/MacOS
platforms.
Most of the ant_globs used are explicitly and knowingly on build directory
(ie. javadoc, jar re) so the warning is quite spurious. The only one that
may be in doubt is the source re one: I added also here because if you use
a code generator (ie. protoc) then it is also correct to glob on builds and
the warning is misleading.
The global value gccdeps was appended to CFLAGS and CXXFLAGS instead of
the actual flags tested against the compiler. This ignored
modifications to the GCCDEPS_FLAGS environment variable and complicated
adding support for additional compilers at the project level.
previously code was erroneously using tg.bld.path instead of tg.path
so for nested wscript calls the wrong directory was used in search.
added also better error handling with error message if an included
directory does not exist
When the 'path' argument was given at TaskGen creation, it was not taken
into account for attributing idx (the path of the build context was).
This is an issue when creating task generators from a waf tool because
their idxs were attributed as if they were in the project root directory,
even if another path was specified, which could lead to output files
collisions.
When CDT is not included in the project (ie. we just have Python and/or Java) the current implementation would not create automatically a call to waf
for the build stage. This patch adds in such cases an external builder that automates the call to waf without the need to manually configure one.
added support to search and add into source path also generated source
files for both java and python. this is useful when using generated code
(ie. protoc and pyqt5) so browsing in eclipse works correclty adding also
paths where generated code is done.
extended example in playground demostrating generated code
If any weights (i.e. `weight` or `tree_weight`) are set on a swig task
then those weights are passed on to the task created to compile the
wrapper generated by swig.
The previous logic in #1709 made an incorrect assumption that the
filename of shared/static library indicates that it was build as
multi-threaded or single threaded. This assumption does not hold in many
Linux distributions.
In addition to that. Boost.Thread and Boost.Log require -pthread (or
some other) flags in order to properly link.
When shared library compiled with precompiled headers enabled, this
change prevents precompiled headers to activate on dependent targets.
Otherwise, there is an issue with -fPIC flag propagation.
not sure if in very old nvcc version this was working anyway but as far as I can read on the documentation this should be the correct way and I tested it out
Waf is a Python-based framework for configuring, compiling and installing applications. Here are perhaps the most important features of Waf:
* *Automatic build order*: the build order is computed from input and output files, among others
* *Automatic dependencies*: tasks to execute are detected by hashing files and commands
* *Performance*: tasks are executed in parallel automatically, the startup time is meant to be fast (separation between configuration and build)
* *Flexibility*: new commands and tasks can be added very easily through subclassing, bottlenecks for specific builds can be eliminated through dynamic method replacement
* *Extensibility*: though many programming languages and compilers are already supported by default, many others are available as extensions
* *IDE support*: Eclipse, Visual Studio and Xcode project generators (waflib/extras/)
* *Documentation*: the application is based on a robust model documented in [The Waf Book](https://waf.io/book/) and in the [API docs](https://waf.io/apidocs/)
* *Python compatibility*: cPython 2.5 to 3.4, Jython 2.5, IronPython, and Pypy
* *Automatic build order*: the build order is computed from input and output files, among others
* *Automatic dependencies*: tasks to execute are detected by hashing files and commands
* *Performance*: tasks are executed in parallel automatically, the startup time is meant to be fast (separation between configuration and build)
* *Flexibility*: new commands and tasks can be added very easily through subclassing, bottlenecks for specific builds can be eliminated through dynamic method replacement
* *Extensibility*: though many programming languages and compilers are already supported by default, many others are available as extensions
* *IDE support*: Eclipse, Visual Studio and Xcode project generators (`waflib/extras/`)
* *Documentation*: the application is based on a robust model documented in [The Waf Book](https://waf.io/book/) and in the [API docs](https://waf.io/apidocs/)
* *Python compatibility*: cPython 2.7 to 3.x, Jython 2.7 and PyPy
Waf is used in particular by innovative companies such as [Avalanche Studios](http://www.avalanchestudios.se) and by open-source projects such as [RTEMS](https://www.rtems.org/). Learn more about Waf by reading [The Waf Book](https://waf.io/book/).
Learn more about Waf by reading [The Waf Book](https://waf.io/book/). For researchers and build system writers, Waf also provides a framework and examples for creating [custom build systems](https://gitlab.com/ita1024/waf/tree/master/build_system_kit) and [package distribution systems](https://gitlab.com/ita1024/waf/blob/master/playground/distnet/README.rst).
For researchers and build system writers, Waf also provides a framework for creating [custom build systems](https://github.com/waf-project/waf/tree/master/build_system_kit) and [package distribution systems](https://github.com/waf-project/waf/tree/master/playground/distnet/README.rst).
Download the project from our page on [waf.io](https://waf.io/) or from a mirror on [freehackers.org](http://www.freehackers.org/~tnagy/release/), consult the [manual](https://waf.io/book/), the [API documentation](https://waf.io/apidocs/) and the [showcases](https://github.com/waf-project/waf/tree/master/demos) and [experiments](https://github.com/waf-project/waf/tree/master/playground).
Download the project from our page on [waf.io](https://waf.io/), consult the [manual](https://waf.io/book/), the [API documentation](https://waf.io/apidocs/) and the [showcases](https://gitlab.com/ita1024/waf/tree/master/demos) and [experiments](https://gitlab.com/ita1024/waf/tree/master/playground).
## HOW TO CREATE THE WAF SCRIPT
Python >= 2.6 is required to generate the waf script, and the resulting file can then run on Python 2.5.
Just run:
```sh
$ ./waf-light configure build
```
Or, if several python versions are installed:
```sh
$ python3 ./waf-light configure build
python ./waf-light configure build
```
## CUSTOMIZATION
The Waf tools in waflib/extras are not added to the waf script. To add
some of them, use the --tools switch. An absolute path can be passed
if the module does not exist under the 'extras' folder:
```sh
$ ./waf-light --tools=swig
./waf-light --tools=swig
```
To customize the initialization, pass the parameter 'prelude'. Here is for example
how to create a waf file using the compat15 module:
The following tools provide support for specific compilers or configurations. More tools are present in the extras_ folder, although they are not documented and as stable as the default tools.
@ -24,7 +24,7 @@ In general, a project will consist of several phases:
Each phase is modelled in the wscript file as a python function which takes as argument an instance of :py:class:`waflib.Context.Context`.
Let's start with a new wscript file in the directory '/tmp/myproject'::
def configure(conf):
def configure(cnf):
print("configure!")
def build(bld):
@ -62,7 +62,7 @@ by using the *${}* symbol, which reads the values from the attribute bld.env::
bld(rule='echo ${MESSAGE}', always=True)
The bld object is an instance of :py:class:`waflib.Build.BuildContext`, its *env* attribute is an instance :py:class:`waflib.ConfigSet.ConfigSet`.
The values are set in this object to be shared/stored/loaded easily. Here is how to do the same thing by sharing data between the configuration and build::
This object is also accessible as an attribute on the `configure()` method's `cnf` parameter. Therefore, values can be shared/stored/loaded easily:
def configure(cnf):
cnf.env.MESSAGE = 'Hello, world!'
@ -111,9 +111,9 @@ Here is a script for a more complicated project::
The method :py:func:`waflib.Tools.c_config.check` executes a build internally to check if the library ``libm`` is present on the operating system.
It will then define variables such as:
* ``conf.env.LIB_M = ['m']``
* ``conf.env.CFLAGS_M = ['-Wall']``
* ``conf.env.DEFINES_M = ['var=foo']``
* ``cnf.env.LIB_M = ['m']``
* ``cnf.env.CFLAGS_M = ['-Wall']``
* ``cnf.env.DEFINES_M = ['var=foo']``
By stating ``use=['M', 'mylib']``, the program *app* is going to inherit all the *M* variables defined
during the configuration. The program will also use the library *mylib* and both the build order and the dependencies
@ -165,7 +165,7 @@ The declaration can be made more user-friendly by binding new methods to the con
Clang-CL is a drop-in MSVC compatible driver replacing CL.exe
The clang compiler offers high compatibility with MSVC, but also offers more up to date C++ support.
It features better code generation, but still adheres to the MSVC ABI, letting you link with link.exe, offering you superior performance but also PDB debug info.
On Windows this waf module should just work, on Linux it tries to find the LLVM replacements and requires an environment containing the paths defined by the vsvars batch files (Visual Studio C++ Developer command prompt).
# Cross compilation
To cross compile for Windows from Linux, you will require the following:
* A partition with Windows installed (NTFS).
* Visual Studio (Tested with 2017).
* The Windows SDK.
* lowntfs-3g file system driver.
Make sure the Windows partition is mounted with `-t lowntfs-3g -o defaults,ignore_case,windows_names`.
This will allow Clang to find all headers and libraries referenced by scripts and headers, otherwise you will run into case sensitivity errors.
You can run a script to make all filenames lowercase, but that edits your Visual Studio installation, and I don't know if that has an effect on upgradability.
Clang uses the following environment variables to detect the Visual Studio install: `VCINSTALLDIR`, `VCToolsInstallDir`, `INCLUDE`, `LIB`, `LIBPATH`
I just copied these from the output of the `set` command in an MSVC command prompt on Windows and translated the paths to Linux paths.
Notice how the semicolon is still used as a path separator.
See `example_environment_linux.sh` for how my setup looks like. It expects the Windows partition to be mounted on `/mnt/windows`, with VS2017 installed and Windows 10 SDK 10.0.17763.0.
To specify a custom LLVM installation, you can put the path in the `LLVM_PATH` environment variable, or put the path in `cfg.env.LLVM_PATH` in your wscript.