c_preproc

C/C++ preprocessor for finding dependencies

Reasons for using the Waf preprocessor by default

  1. Some c/c++ extensions (Qt) require a custom preprocessor for obtaining the dependencies (.moc files)

  2. Not all compilers provide .d files for obtaining the dependencies (portability)

  3. A naive file scanner will not catch the constructs such as “#include foo()”

  4. A naive file scanner will catch unnecessary dependencies (change an unused header -> recompile everything)

Regarding the speed concerns:

  • the preprocessing is performed only when files must be compiled

  • the macros are evaluated only for #if/#elif/#include

  • system headers are not scanned by default

Now if you do not want the Waf preprocessor, the tool +gccdeps* uses the .d files produced during the compilation to track the dependencies (useful when used with the boost libraries). It only works with gcc >= 4.4 though.

A dumb preprocessor is also available in the tool c_dumbpreproc

exception waflib.Tools.c_preproc.PreprocError(msg='', ex=None)[source]
__annotations__ = {}
__firstlineno__ = 32
__static_attributes__ = ()
waflib.Tools.c_preproc.POPFILE = '-'

Constant representing a special token used in waflib.Tools.c_preproc.c_parser.start() iteration to switch to a header read previously

waflib.Tools.c_preproc.recursion_limit = 150

Limit on the amount of files to read in the dependency scanner

waflib.Tools.c_preproc.go_absolute = False

Set to True to track headers on files in /usr/include, else absolute paths are ignored (but it becomes very slow)

waflib.Tools.c_preproc.use_trigraphs = 0

Apply trigraph rules (False by default)

waflib.Tools.c_preproc.g_optrans = {'and': '&&', 'and_eq': '&=', 'bitand': '&', 'bitor': '|', 'compl': '~', 'not': '!', 'not_eq': '!', 'or': '||', 'or_eq': '|=', 'xor': '^', 'xor_eq': '^='}

Operators such as and/or/xor for c++. Set an empty dict to disable.

waflib.Tools.c_preproc.re_lines = re.compile('^[ \t]*(?:#|%:)[ \t]*(ifdef|ifndef|if|else|elif|endif|include|import|define|undef|pragma)[ \t]*(.*)\r*$', re.IGNORECASE|re.MULTILINE)

Match #include lines

waflib.Tools.c_preproc.re_mac = re.compile('^[a-zA-Z_]\\w*')

Match macro definitions

waflib.Tools.c_preproc.re_fun = re.compile('^[a-zA-Z_][a-zA-Z0-9_]*[(]')

Match macro functions

waflib.Tools.c_preproc.re_pragma_once = re.compile('^\\s*once\\s*', re.IGNORECASE)

Match #pragma once statements

waflib.Tools.c_preproc.re_nl = re.compile('\\\\\r*\n', re.MULTILINE)

Match newlines

waflib.Tools.c_preproc.re_cpp = re.compile('//.*?$|/\\*.*?\\*/|\\\'(?:\\\\.|[^\\\\\\\'])*\\\'|"(?:\\\\.|[^\\\\"])*"', re.MULTILINE|re.DOTALL)

Filter C/C++ comments

waflib.Tools.c_preproc.trig_def = [('??=', '#'), ('??-', '~'), ('??/', '\\'), ('??!', '|'), ("??'", '^'), ('??(', '['), ('??)', ']'), ('??<', '{'), ('??>', '}')]

Trigraph definitions

waflib.Tools.c_preproc.chr_esc = {"'": 39, '0': 0, '\\': 92, 'a': 7, 'b': 8, 'f': 11, 'n': 10, 'r': 13, 't': 9, 'v': 12}

Escape characters

waflib.Tools.c_preproc.NUM = 'i'

Number token

waflib.Tools.c_preproc.OP = 'O'

Operator token

waflib.Tools.c_preproc.IDENT = 'T'

Identifier token

waflib.Tools.c_preproc.STR = 's'

String token

waflib.Tools.c_preproc.CHAR = 'c'

Character token

waflib.Tools.c_preproc.tok_types = ['i', 's', 'T', 'O']

Token types

waflib.Tools.c_preproc.exp_types = ["0[xX](?P<hex>[a-fA-F0-9]+)(?P<qual1>[uUlL]*)|L*?'(?P<char>(\\\\.|[^\\\\'])+)'|(?P<n1>\\d+)[Ee](?P<exp0>[+-]*?\\d+)(?P<float0>[fFlL]*)|(?P<n2>\\d*\\.\\d+)([Ee](?P<exp1>[+-]*?\\d+))?(?P<float1>[fFlL]*)|(?P<n4>\\d+\\.\\d*)([Ee](?P<exp2>[+-]*?\\d+))?(?P<float2>[fFlL]*)|(?P<oct>0*)(?P<n0>\\d+)(?P<qual2>[uUlL]*)", 'L?"([^"\\\\]|\\\\.)*"', '[a-zA-Z_]\\w*', '%:%:|<<=|>>=|\\.\\.\\.|<<|<%|<:|<=|>>|>=|\\+\\+|\\+=|--|->|-=|\\*=|/=|%:|%=|%>|==|&&|&=|\\|\\||\\|=|\\^=|:>|!=|##|[\\(\\)\\{\\}\\[\\]<>\\?\\|\\^\\*\\+&=:!#;,%/\\-\\?\\~\\.]']

Expression types

waflib.Tools.c_preproc.re_clexer = re.compile('(?P<i>0[xX](?P<hex>[a-fA-F0-9]+)(?P<qual1>[uUlL]*)|L*?\'(?P<char>(\\\\.|[^\\\\\'])+)\'|(?P<n1>\\d+)[Ee](?P<exp0>[+-]*?\\d+)(?P<float0>[fFlL]*)|(?P<n2>\\d*\\.\\d+)([Ee](?P<exp1>[+-]*?\\d+))?(?P<float1, re.MULTILINE)

Match expressions into tokens

waflib.Tools.c_preproc.accepted = 'a'

Parser state is accepted

waflib.Tools.c_preproc.ignored = 'i'

Parser state is ignored, for example preprocessor lines in an #if 0 block

waflib.Tools.c_preproc.undefined = 'u'

Parser state is undefined at the moment

waflib.Tools.c_preproc.skipped = 's'

Parser state is skipped, for example preprocessor lines in a #elif 0 block

waflib.Tools.c_preproc.repl(m)[source]

Replace function used with waflib.Tools.c_preproc.re_cpp

waflib.Tools.c_preproc.prec = {'!=': 4, '%': 0, '&': 5, '&&': 6, '*': 0, '+': 1, ',': 7, '-': 1, '/': 0, '<': 3, '<<': 2, '<=': 3, '==': 4, '>': 3, '>=': 3, '>>': 2, '^': 5, '|': 5, '||': 6}

Operator precedence rules required for parsing expressions of the form:

#if 1 && 2 != 0
waflib.Tools.c_preproc.reduce_nums(val_1, val_2, val_op)[source]

Apply arithmetic rules to compute a result

Parameters:
  • val1 (int or string) – input parameter

  • val2 (int or string) – input parameter

  • val_op (string) – C operator in +, /, -, etc

Return type:

int

waflib.Tools.c_preproc.get_num(lst)[source]

Try to obtain a number from a list of tokens. The token types are defined in waflib.Tools.ccroot.tok_types.

Parameters:

lst (list of tuple (tokentype, value)) – list of preprocessor tokens

Returns:

a pair containing the number and the rest of the list

Return type:

tuple(value, list)

waflib.Tools.c_preproc.get_term(lst)[source]

Evaluate an expression recursively, for example:

1+1+1 -> 2+1 -> 3
Parameters:

lst (list of tuple(token, value)) – list of tokens

Returns:

the value and the remaining tokens

Return type:

value, list

waflib.Tools.c_preproc.reduce_eval(lst)[source]

Take a list of tokens and output true or false for #if/#elif conditions.

Parameters:

lst (list of tuple(token, value)) – a list of tokens

Returns:

a token

Return type:

tuple(NUM, int)

waflib.Tools.c_preproc.stringize(lst)[source]

Merge a list of tokens into a string

Parameters:

lst (list of tuple(token, value)) – a list of tokens

Return type:

string

waflib.Tools.c_preproc.paste_tokens(t1, t2)[source]

Token pasting works between identifiers, particular operators, and identifiers and numbers:

a ## b  ->  ab
> ## =  ->  >=
a ## 2  ->  a2
Parameters:
  • t1 (tuple(type, value)) – token

  • t2 (tuple(type, value)) – token

waflib.Tools.c_preproc.reduce_tokens(lst, defs, ban=[])[source]

Replace the tokens in lst, using the macros provided in defs, and a list of macros that cannot be re-applied

Parameters:
  • lst (list of tuple(token, value)) – list of tokens

  • defs (dict) – macro definitions

  • ban (list of string) – macros that cannot be substituted (recursion is not allowed)

Returns:

the new list of tokens

Return type:

value, list

waflib.Tools.c_preproc.eval_macro(lst, defs)[source]

Reduce the tokens by waflib.Tools.c_preproc.reduce_tokens() and try to return a 0/1 result by waflib.Tools.c_preproc.reduce_eval().

Parameters:
  • lst (list of tuple(token, value)) – list of tokens

  • defs (dict) – macro definitions

Return type:

int

waflib.Tools.c_preproc.extract_macro(txt)[source]
Process a macro definition of the form::

#define f(x, y) x * y

into a function or a simple macro without arguments

Parameters:

txt (string) – expression to exact a macro definition from

Returns:

a tuple containing the name, the list of arguments and the replacement

Return type:

tuple(string, [list, list])

waflib.Tools.c_preproc.extract_include(txt, defs)[source]

Process a line in the form:

#include foo
Parameters:
  • txt (string) – include line to process

  • defs (dict) – macro definitions

Returns:

the file name

Return type:

string

waflib.Tools.c_preproc.parse_char(txt)[source]

Parse a c character

Parameters:

txt (string) – character to parse

Returns:

a character literal

Return type:

string

waflib.Tools.c_preproc.tokenize(s)[source]

Convert a string into a list of tokens (shlex.split does not apply to c/c++/d)

Parameters:

s (string) – input to tokenize

Returns:

a list of tokens

Return type:

list of tuple(token, value)

class waflib.Tools.c_preproc.c_parser(nodepaths=None, defines=None)[source]

Used by waflib.Tools.c_preproc.scan() to parse c/h files. Note that by default, only project headers are parsed.

__annotations__ = {}
__dict__ = mappingproxy({'__module__': 'waflib.Tools.c_preproc', '__firstlineno__': 797, '__doc__': '\nUsed by :py:func:`waflib.Tools.c_preproc.scan` to parse c/h files. Note that by default,\nonly project headers are parsed.\n', '__init__': <function c_parser.__init__>, 'cached_find_resource': <function c_parser.cached_find_resource>, 'tryfind': <function c_parser.tryfind>, 'filter_comments': <function c_parser.filter_comments>, 'parse_lines': <function c_parser.parse_lines>, 'addlines': <function c_parser.addlines>, 'start': <function c_parser.start>, 'define_name': <function c_parser.define_name>, '__static_attributes__': ('ban_includes', 'count_files', 'curfile', 'current_file', 'currentnode_stack', 'defs', 'lines', 'listed', 'names', 'nodepaths', 'nodes', 'state'), '__dict__': <attribute '__dict__' of 'c_parser' objects>, '__weakref__': <attribute '__weakref__' of 'c_parser' objects>, '__annotations__': {}})
__firstlineno__ = 797
__static_attributes__ = ('ban_includes', 'count_files', 'curfile', 'current_file', 'currentnode_stack', 'defs', 'lines', 'listed', 'names', 'nodepaths', 'nodes', 'state')
lines

list of lines read

nodepaths

Include paths

nodes

List of waflib.Node.Node found so far

names

List of file names that could not be matched by any file

curfile

Current file

ban_includes

Includes that must not be read (#pragma once)

listed

Include nodes/names already listed to avoid duplicates in self.nodes/self.names

cached_find_resource(node, filename)[source]

Find a file from the input directory

Parameters:
Returns:

the node if found, or None

Return type:

waflib.Node.Node

tryfind(filename, kind='"', env=None)[source]

Try to obtain a node from the filename based from the include paths. Will add the node found to waflib.Tools.c_preproc.c_parser.nodes or the file name to waflib.Tools.c_preproc.c_parser.names if no corresponding file is found. Called by waflib.Tools.c_preproc.c_parser.start.

Parameters:

filename (string) – header to find

Returns:

the node if found

Return type:

waflib.Node.Node

filter_comments(node)[source]

Filter the comments from a c/h file, and return the preprocessor lines. The regexps waflib.Tools.c_preproc.re_cpp, waflib.Tools.c_preproc.re_nl and waflib.Tools.c_preproc.re_lines are used internally.

Returns:

the preprocessor directives as a list of (keyword, line)

Return type:

a list of string pairs

addlines(node)[source]

Add the lines from a header in the list of preprocessor lines to parse

Parameters:

node (waflib.Node.Node) – header

start(node, env)[source]

Preprocess a source file to obtain the dependencies, which are accumulated to waflib.Tools.c_preproc.c_parser.nodes and waflib.Tools.c_preproc.c_parser.names.

Parameters:
define_name(line)[source]
Parameters:

line (string) – define line

Return type:

string

Returns:

the define name

waflib.Tools.c_preproc.scan(task)[source]

Get the dependencies using a c/c++ preprocessor, this is required for finding dependencies of the kind:

#include some_macro()

This function is bound as a task method on waflib.Tools.c.c and waflib.Tools.cxx.cxx for example