gcc/libcody

libCODY: COmpiler DYnamism¹

Copyright (C) 2020 Nathan Sidwell, nathan@acm.org

libCODY is an implementation of a communication protocol between compilers and build systems.

WARNING: This is preliminary software.

In addition to supporting C++modules, this may also support LTO requirements and could also deal with generated #include files and feed the compiler with prepruned include paths and whatnot. (The system calls involved in include searches can be quite expensive on some build infrastructures.)

• Client and Server objects
• Direct connection for in-process use
• Testing with Joust (that means nothing to you, doesn't it!)

Problem Being Solved

The origin is in C++20 modules:

import foo;

At that import, the compiler needs² to load up the compiled serialization of module foo. Where is that file? Does it even exist? Unless the build system already knows the dependency graph, this might be a completely unknown module. Now, the build system knows how to build things, but it might not have complete information about the dependencies. The ultimate source of dependencies is the source code being compiled, and specifying the same thing in multiple places is a recipe for build skew.

Hence, a protocol by which a compiler can query a build system. This was originally described in p1184r1:A Module Mapper. Along with a proof-of-concept hack in GNUmake, described in p1602:Make Me A Module. The current implementation has evolved and an update to p1184 will be forthcoming.

Packet Encoding

The protocol is turn-based. The compiler sends a block of one or more requests to the builder, then waits for a block of responses to all of those requests. If the builder needs to compile something to satisfy a request, there may be some time before the response. A builder may service multiple compilers concurrently, each as a separate connection.

When multiple requests are in a block, the responses are also in a block, and in corresponding order. The responses must not be commenced eagerly -- they must wait until the incoming block has ended (as mentioned above, it is turn-based). To do otherwise risks deadlock, as there is no requirement for a sending end of the communication to listen for incoming responses (or new requests) until it has completed sending its current block.

Every request has a response.

Requests and responses are user-readable text. It is not intended as a transmission medium to send large binary objects (such as compiled modules). It is presumed the builder and the compiler share a file system, for that kind of thing.³

Messages characters are encoded in UTF8.

Messages are a sequence of octets ending with a NEWLINE (0xa). The lines consist of a sequence of words, separated by WHITESPACE (0x20 or 0x9). Words themselves do not contain WHITESPACE. Lines consisting solely of WHITESPACE (or empty) are ignored.

To encode a block of multiple messages, non-final messages end with a single word of SEMICOLON (0x3b), immediately before the NEWLINE. Thus a serial connection can determine whether a block is complete without decoding the messages.

Words containing characters in the set [-+_/%.A-Za-z0-9] need not be quoted. Words containing characters outside that set should be quoted. A zero-length word may be achieved with ''

Quoted words begin and end with APOSTROPHE (x27). Within the quoted word, BACKSLASH (x5c) is used as an escape mechanism, with the following meanings:

• \n - NEWLINE (0xa)
• \t - TAB (0x9)
• \' - APOSTROPHE (')
• \\ - BACKSLASH (\)

Characters in the range [0x00, 0x20) and 0x7f are encoded with one or two lowercase hex characters. Octets in the range [0x80,0xff) are UTF8 encodings of unicode characters outside the traditional ASCII set and passed as such.

Decoding should be more relaxed. Unquoted words containing characters in the range [0x20,0xff] other than BACKSLASH or APOSTROPHE should be accepted. In a quoted sequence, \ followed by one or two lower case hex characters decode to that octet. Further, words can be constructed from a mixture of abutted quoted and unquoted sequences. For instance FOO' 'bar would decode to the word FOO bar.

Notice that the block continuation marker of ; is not a valid encoding of the word ;, which would be ';'.

It is recommended that words are separated by single SPACE characters.

Messages

The message descriptions use $metavariable examples.

The request messages are specific to a particular action. The response messages are more generic, describing their value types, but not their meaning. Message consumers need to know the response to decode them. Notice the Packet::GetRequest() method records in response packets what the request being responded to was. Do not confuse this with the Packet::GetCode () method.

Responses

The simplest response is a single:

OK

This indicates the request was successful.

An error response is:

ERROR $message

The message is a human-readable string. It indicates failure of the request.

Pathnames are encoded with:

PATHNAME $pathname

Boolean responses use:

BOOL (TRUE|FALSE)

Handshake Request

The first message is a handshake:

HELLO $version $compiler $ident

The $version is a numeric value, currently 1. $compiler identifies the compiler — builders may need to keep compiled modules from different compilers separate. $ident is an identifier the builder might use to identify the compilation it is communicating with.

Responses are:

HELLO $version $builder [$flags]

A successful handshake. The communication is now connected and other messages may be exchanged. An ERROR response indicates an unsuccessful handshake. The communication remains unconnected.

There is nothing restricting a handshake to its own message block. Of course, if the handshake fails, subsequent non-handshake messages in the block will fail (producing error responses).

The $flags word, if present allows a server to control what requests might be given. See below.

C++ Module Requests

A set of requests are specific to C++ modules:

Flags

Several requests and one response have an optional $flags word. These are the Cody::Flags value pertaining to that request. If omitted the value 0 is implied. The following flags are available:

• 0, None: No flags.

• 1<<0, NameOnly: The request is for the name only, and not the CMI contents.

The NameOnly flag may be provded in a handshake response, and indicates that the server is interested in requests only for their implied dependency information. It may be provided on a request to indicate that only the CMI name is required, not its contents (for instance, when preprocessing). Note that a compiler may still make NameOnly requests even if the server did not ask for such.

Repository

All relative CMI file names are relative to a repository. (There are usually no absolute CMI files). The repository may be determined with:

MODULE-REPO

A PATHNAME response is expected. The $pathname may be an empty word, which is equivalent to .. When the response is a relative pathname, it must be relative to the client's current working directory (which might be a process on a different host to the server). You may set the repository to /, if you with to use paths relative to the root directory.

Exporting

A compilation of a module interface, partition or header unit can inform the builder with:

MODULE-EXPORT $module [$flags]

This will result in a PATHNAME response naming the Compiled Module Interface pathname to write.

The MODULE-EXPORT request does not indicate the module has been successfully compiled. At most one MODULE-EXPORT is to be made, and as the connection is for a single compilation, the builder may infer dependency relationships between the module being generated and import requests made.

Named module names and header unit names are distinguished by making the latter unambiguously look like file names. Firstly, they must be fully resolved according to the compiler's usual include path. If that results in an absolute name file name (beginning with /, or certain other OS-specific sequences), all is well. Otherwise a relative file name must be prefixed by ./ to be distinguished from a similarly named named module. This prefixing must occur, even if the header-unit's name contains characters that cannot appear in a named module's name.

It is expected that absolute header-unit names convert to relative CMI names, to keep all CMIs within the CMI repository. This means that steps must be taken to distinguish the CMIs for /here from ./here, and this can be achieved by replacing the leading ./ directory with ,/, which is visually similar but does not have the self-reference semantics of dot. Likewise, header-unit names containing .. directories, can be remapped to ,,. (When symlinks are involved bob/dob/.. might not be bob, of course.) C++ header-unit semantics are such that there is no need to resolve multiple ways of spelling a particular header-unit to a unique CMI file.

Successful compilation of an interface is indicated with a subsequent:

MODULE-COMPILED $module [$flags]

request. This indicates the CMI file has been written to disk, so that any other compilations waiting on it may proceed. Depending on compiler implementation, the CMI may be written before the compilation completes. A single OK response is expected.

Compilation failure can be inferred by lack of a MODULE-COMPILED request. It is presumed the builder can determine this, as it is also responsible for launching and reaping the compiler invocations themselves.

Importing

Importation, including that of header-units, uses:

MODULE-IMPORT $module [$flags]

A PATHNAME response names the CMI file to be read. Should the builder have to invoke a compilation to produce the CMI, the response should be delayed until that occurs. If such a compilation fails, an error response should be provided to the requestor — which will then presumably fail in some manner.

Include Translation

Include translation can be determined with:

INCLUDE-TRANSLATE $header [$flags]

The header name, $header, is the fully resolved header name, in the above-mentioned unambiguous filename form. The response will either be a BOOL response indicating textual inclusion, or a PATHNAME response naming the CMI for such translation. The BOOL value is TRUE, if the header is known to be a textual header, and FALSE if nothing is known about it -- the latter might cause diagnostics about incomplete knowledge.

GCC LTO Messages

These set of requests are used for GCC LTO jobserver integration with GNU Make

Building libCody

Libcody is written in C++11. (It's a intended for compilers, so there'd be a bootstrapping problem if it used the latest and greatest.)

Using configure and make.

It supports the usual configure, make, make check & make install sequence. It does not support building in the source directory -- that just didn't drop out, and it's not how I build things (because, again, for compilers). Excitingly it uses my own joust test harness, so you'll need to build and install that somewhere, if you want the comfort of testing.

The following configure options are available, in addition to the usual set:

• --enable-checking Compile with assert-like checking. Defaults to on.

• --with-tooldir=DIR Prepend DIR to PATH when building (DIR need not already include the trailing /bin, and the right things happen). Use this if you need to point to non-standard tools that you usually don't have in your path. This path is also used when the configure script searches for programs.

• --with-toolinc=DIR, --with-toollib=DIR, include path and library path variants of --with-tooldir. If these are siblings of the tool bin directory, they'll be found automatically.

• --with-compiler=NAME Specify a particular compiler to use. Usually what configure finds is sufficiently usable.

• --with-bugurl=URL Override the bugreporting URL. Do this if you're providing libcody as part of a package that /you/ are supporting.

• --enable-maintainer-mode Specify that rules to rebuild things like configure (with autoconf) should be enabled. When not enabled, you'll get a message if these appear out of date, but that can happen naturally after an update or clone as git, in common with other VCs, doesn't preserve the relative ordering of file modifications. You can use make MAINTAINER=touch to shut make up, if this occurs (or manually execute the autoconf and related commands).

When building, you can override the default optimization flags with CXXFLAGS=$flags. I often build a debuggable library with make CXXFLAGS=-g3.

The Makefile will also parallelize according to the number of CPUs, unless you specify explicitly with a -j option. This is a little clunky, as it's not possible to figure out inside the makefile whether the user provided -j. (Or at least I've not figured out how.)

Using cmake and make

In the clang/LLVM project

The primary motivation for a cmake implementation is to allow building libcody "in tree" in clang/LLVM. In that case, a checkout of libcody can be placed (or symbolically linked) into clang/tools. This will configure and build the library along with other LLVM dependencies.

NOTE This is not treated as an installable entity (it is present only for use by the project).

NOTE The testing targets would not be appropriate in this configuration; it is expected that lit-based testing of the required functionality will be done by the code using the library.

Stand-alone

For use on platforms that don't support configure & make effectively, it is possible to use the cmake & make process in stand-alone mode (similar to the configure & make process above).

An example use.

cmake -DCMAKE_INSTALL_PREFIX=/path/to/installation -DCMAKE_CXX_COMPILER=clang++ /path/to/libcody/source
make
make install

Supported flags (additions to the usual cmake ones).

• -DCODY_CHECKING=ON,OFF: Compile with assert-like checking. (defaults ON)

• -DCODY_WITHEXCEPTIONS=ON,OFF: Compile with C++ exceptions and RTTI enabled. (defaults OFF, to be compatible with GCC and LLVM).

TODO: At present there is no support for ctest integration (this should be feasible, provided that joust is installed and can be discovered by cmake).

API

The library defines entities in the ::Cody namespace.

There are 4 user-visible classes:

• Packet: Responses to requests are Packets. These have a code, indicating the response kind, and a payload.

• Client: The compiler-end of a connection. Requests may be made and responses are returned.

• Server: The builder-end of a connection. Requests may be waited for, and responses made. Builders that serve multiple concurrent connections and spawn compilations to resolve dependencies may need to derive from this class to provide response queuing.

• Resolver: The processing engine of the builder side. User code is expected to derive from this class and provide virtual function overriders to affect the semantics of the resolver.

In addition there are a number of helpers to setup connections.

Logically the Client and the Server communicate via a sequential channel. The channel may be provided by:

• two pipes, with different file descriptors for reading and writing at each end.

• a socket, which will use the same file descriptor for reading and writing. the socket can be created in a number of ways, including Unix domain and IPv6 TCP, for which helpers are provided.

• a direct, in-process, connection, using buffer swapping.

The communication channel is presumed reliable.

Refer to the (currently very sparse) doxygen-generated documentation for details of the API.

Examples

To create an in-process resolver, use the following boilerplate:

class MyResolver : Cody::Resolver { ... stuff here ... };

Cody::Client *MakeClient (char const *maybe_ident)
{
  auto *r = new MyResolver (...);
  auto *s = new Cody::Server (r);
  auto *c = new Cody::Client (s);

  auto t = c->ConnectRequest ("ME", maybe_ident);
  if (t.GetCode () == Cody::Client::TC_CONNECT)
    ;// Yay!
  else if (t.GetCode () == Cody::Client::TC_ERROR)
    report_error (t.GetString ());

  return c;
}

For a remotely connecting client:

Cody::Client *MakeClient ()
{
  char const *err = nullptr;
  int fd = OpenInet6 (char const **err, name, port);
  if (fd < 0)
    { ... error... return nullptr;}

  auto *c = new Cody::Client (fd);

  auto t = c->ConnectRequest ("ME", maybe_ident);
  if (t.GetCode () == Cody::Client::TC_CONNECT)
    ;// Yay!
  else if (t.GetCode () == Cody::Client::TC_ERROR)
    report_error (t.GetString ());

  return c;
}

Future Directions

• Current Directory. There is no mechanism to check the builder and the compiler have the same working directory. Perhaps that should be addressed.

• Include path canonization and/or header file lookup. This can be expensive, particularly with many -I options, due to the system calls. Perhaps using a common resource would be cheaper?

• Generated header file lookup/construction. This is essentially the same problem as importing a module, and build systems are crap at dealing with this.

• Link-time compilations. Another place the compiler would like to ask the build system to do things.

• C++20 API entrypoints — std:string_view would be nice

• Exception-safety audit. Exceptions are not used, but memory exhaustion could happen. And perhaps user's resolver code employs exceptions?

1: Or a small town in Wyoming

2: This describes one common implementation technique. The std itself doesn't require such serializations, but the ability to create them is kind of the point. Also, 'compiler' is used where we mean any consumer of a module, and 'build system' where we mean any producer of a module.

3: Even when the builder is managing a distributed set of compilations, the builder must have a mechanism to get source files to, and object files from, the compilations. That scheme can also transfer the CMI files.