Embedding Files

Features

• Embed files in ELF objects using objcopy.
• x86-64-only support.
• Embed a set of files (including possibly generated files) as a single blob. Note: This creates an intermediate file when building, you should decide if this is acceptable for your use-case (for very large files this may not be).
• Embeds an associated index file that records file path, offset and size to file in blob.

Configuration

First load tool, nominally in the top-level wscript:

def configure(cnf):
    cnf.wtools_load("embed")

Then the following taskgen attributes with expected type and default value (indicated with (<type>) and [<default>]) controls the feature:

• embed_target (str): Used in generated symbol names and intermediate files.
• embed_source (stringlist/nodelist): Files to embed.
• embed_dest (str) [""] : Destination path used.
• embed_relative_trick (bool) [False]: Whether to record paths as seen from embed_relative_base.
• embed_relative_base (str/node) [bld.path]: The embed root which determines the recorded relative path to each embed_source. Default is to use the taskgen path. note: If source file is in the build directory the relative path will be recorded as relative to the corresponding path in build dir.
• embed_section (str) [".rodata,alloc,load,readonly,data,contents"]: Section and flags as expected by objcopy --rename-section to place embedded files. Default is the large read only data section ".lrodata".
• embed_gnustack (bool) [True]: When true a .note.GNU-stack section will be added to generated objects, which prevents ld from assuming that stack must be made executable.

Note on Embedded File Paths

To facilitate use both in simple flat lookup or as a sort of embedded filesystem the paths of embedded files can:

• Be flattened if embed_relative_trick=False.

  With e.g. embed_dest="/root" and embed_source="src/file.txt" this results in the path /root/file.txt

• Record with relative component if embed_relative_trick=True.

  With e.g. embed_dest="/root" and embed_source="src/file.txt" this results in the path /root/src/file.txt. The relative base can be configured with embed_relative_base.

Note: In both cases the paths are normalized so e.g. /root/../file.txt yields /file.txt

Details

Objects and Symbols

This feature will produce the following:

1. Two generated files {embed_targe}_index and {embed_target}_data containing an index used to enumerate embedded files and the merged files to be embedded together with file path information.

2. Two ELF object files {embed_target}_data.o and {embed_target}_index.o containing the generated data as well as symbols used to access the data. The object files are added to the waf taskgen attribute compile_tasks which means the object files will be linked in automatically when used in e.g. applications or libraries.

   The (global) linker symbols are:

   • embed_{embed_target}_index_first and embed_{embed_target}_index_last (in `{embed_target}_index.o) which points to beginning and one-past end of index entries. See below for how to read the index.
   • embed_{embed_target}_data (in {embed_target}_data.o) which points to the embedded data. The index is used to access individual files.

Reading the Index

The index contains offets and sizes of file paths and corresponding file contents which is used to to safely access the correct ranges of the symbol embed_{embed_target}_data

Each file has an entry and each entry is equivalent to the following C-struct on x86-64 (i.e. 4 8-byte unsigned integers):

struct IndexEntry {
    /* Offset to beginning of UTF-8 encoded file path [char *] */
    size_t path_first;
 
    /* Size of path in bytes */
    size_t path_size;
 
    /* Offset to beginning of file data [std::byte*] */
    size_t data_first;
 
    /* Size of data in bytes */
    size_t data_size;
};

Index is accessed as an array of IndexEntry in the half-open range [embed_{embed_target}_index_first, embed_{embed_target}_index_last)

Example iterating all entries using embed_target="foo":

extern "C" {
extern IndexEntry const embed_foo_index_first[];
extern IndexEntry const embed_foo_index_last[];
extern char const embed_data_data[];
}
 
// C-style for loop
for (IndexEntry const* entry = embed_foo_index_first; entry < embed_foo_index_last; ++entry) {
  // Do something
}
 
// for_each from <algorithm>
std::for_each(embed_foo_index_first, embed_foo_index_last, [](IndexEntry const &entry) mutable {
  // Do something
});

Usage

C++ example which prints file name and content of the embedded files "src/first.txt" and "src/second.txt" using embed_target="foo":

Example build script:

# wscript
 
def build(bld):
    bld(
        target="example",
        features="cxx cxxprogram embed",
        source="src/main.cpp",
        embed_target="foo",
        embed_relative_trick=True,
        embed_source=[
            "src/first.txt",
            "src/second.txt",
        ],
    )

Example program:

// main.cpp
#include <algorithm>
#include <cstdint>
#include <iostream>
#include <string_view>
 
// Here we use a slightly different way to express the index structure:
struct IndexEntry {
  struct Span {
    std::size_t offset;
    std::size_t size;
  };
  Span path;
  Span data;
};
 
extern "C" {
// Symbols (note: embed_target="foo" was used)
extern IndexEntry const embed_foo_index_first[];
extern IndexEntry const embed_foo_index_last[];
extern char const embed_foo_data[];
}
 
// Example usage

auto MakeView(IndexEntry::Span const &s) -> std::string_view {
  return std::string_view(&embed_foo_data[s.offset], s.size);
};
 
int main() {
  std::for_each(embed_foo_index_first, embed_foo_index_last,
                [](IndexEntry const &entry) {
                  auto path = MakeView(entry.path);
                  auto data = MakeView(entry.data);
                  std::cout << "File path: '" << path
                            << "', file size: " << data.size() << "b\n";
                  std::cout << "File content: \n" << data << '\n';
                });
}