BitBake Build Process: Tasks, Packages and Debugging

The familiar fetch → unpack → patch → configure → compile → install → package diagram is useful, but it hides the fact that matters most when a build behaves strangely: BitBake schedules a dependency graph, not a universal linear script. Tasks can be skipped from stamps, restored from shared state, inserted by classes, or run in parallel when their dependencies allow it.

Quick answer: the common recipe path

do_fetch → do_unpack → do_patch → do_prepare_recipe_sysroot
                          ↓
                     do_configure
                          ↓
                      do_compile
                          ↓
                  do_install (into ${D})
                          ↓
        do_package → do_packagedata / do_package_qa
                          ↓
           do_package_write_rpm|deb|ipk
                          ↓
                       do_build

What this diagram leaves out on purpose

• Classes and recipe metadata add, remove, and order tasks; images, kernels, SDKs, and native tools have additional paths.

• do_build is the normal recipe target and depends on the tasks needed to complete that recipe. It does not mean every task always executes again.

• Setscene tasks can restore eligible outputs from shared state before real tasks are scheduled.

• Inter-recipe dependencies prepare recipe-specific sysroots before configure/compile work.

• Package splitting occurs after files have been installed into ${D}; one recipe can create several binary packages.

BitBake starts with metadata and a graph

BitBake parses configuration, layers, recipes, includes, append files, and inherited classes. It resolves providers and versions, expands variables and overrides, constructs task dependencies, calculates signatures, checks stamps and shared state, then schedules runnable tasks. The build directory is therefore an output of metadata resolution—not the source of truth.

• Recipe (`.bb`): metadata and task implementation for a buildable target.

• Append (`.bbappend`): layer-specific changes applied to a matching recipe.

• Class (`.bbclass`): reusable behavior that can define tasks, functions, flags, and dependencies.

• Task: a shell or Python unit such as do_compile; the task graph orders it using dependencies.

• Recipe name versus package name: ${PN} commonly names the recipe/main output, but PACKAGES can produce ${PN}-dev, ${PN}-dbg, libraries, plugins, and other installable packages.

Names that are easy to mix up

• A target is what you ask BitBake to build; provider resolution chooses the recipe that supplies it.

• A recipe describes how source becomes one or more outputs; it is not an RPM/DEB/IPK itself.

• A task is one executable graph node within recipe processing.

• A package is an installable output assembled from files staged by a recipe.

• An image is a root filesystem and related artifacts composed from selected packages and image metadata.

1. do_fetch: acquire declared inputs

do_fetch uses SRC_URI and the appropriate fetcher to obtain source archives, Git revisions, patches, and local files. Downloads usually land under DL_DIR; network policy, mirrors, checksums, and source revision rules influence whether access occurs.

• Pinned revisions and checksums improve reproducibility; floating branches can change without a recipe edit.

• A successful fetch proves that inputs were acquired, not that licensing, patching, or compilation will succeed.

• Local file:// entries are searched through FILESPATH; they are not fetched from the network.

• Shared DL_DIR content may serve multiple builds, so deleting it is broader than cleaning one work directory.

2. do_unpack and do_patch: create the source tree

do_unpack extracts fetched inputs into the recipe work area. do_patch then applies patch and diff entries selected from SRC_URI. The effective source directory is normally ${S}, which may need adjustment when an archive has an unusual top-level layout.

• Patch order follows resolved metadata and SRC_URI, including append files and overrides.

• A patch failure can mean wrong source revision, incorrect strip level, stale patch context, duplicate application, or unexpected layer priority.

• Inspect ${S} after do_patch when validating the actual tree, but keep durable fixes in a layer—not by editing tmp/work files.

3. do_prepare_recipe_sysroot: stage build dependencies

Before compilation, OpenEmbedded assembles a recipe-specific sysroot from components supplied by recipes in DEPENDS. This replaces the old global staging model and helps prevent one recipe from accidentally seeing undeclared headers or libraries.

4. do_configure: prepare the build system

do_configure prepares Make, Autotools, CMake, Meson, Cargo, or another inherited build system using the cross-compilation environment. It does not always run a literal ./configure; the implementation comes from the recipe and its classes.

• ${S} is the source directory; ${B} is the build directory and may be separate.

• Cross tools, flags, sysroot paths, and feature options are supplied through metadata and class logic.

• Configuration failures often reveal missing DEPENDS, host-contamination assumptions, unsupported options, or an incorrect ${S}/${B}.

5. do_compile: build, but do not package

do_compile invokes the recipe’s build implementation inside BitBake’s controlled environment. A successful compile typically creates binaries in ${B}; those files are not yet arranged into target filesystem paths or split into installable packages.

• Parallelism comes from both BitBake task scheduling and the build tool’s own job flags. Races may disappear when parallelism is reduced, but that is evidence to diagnose—not usually the final fix.

• The compiler is normally a target cross-compiler. Native and nativesdk variants have different sysroots and execution contexts.

• A manual command in a random shell can succeed while the BitBake task fails because its environment, working directory, dependencies, PATH, and flags differ.

6. do_install: populate the package staging tree

do_install copies build results into ${D}, a temporary destination that mirrors target filesystem paths. It runs under fakeroot so ownership metadata can be represented without making the build user root.

do_install() {
    install -d ${D}${bindir}
    install -m 0755 ${B}/example ${D}${bindir}/example
 
    install -d ${D}${sysconfdir}/example
    install -m 0644 ${WORKDIR}/example.conf \
        ${D}${sysconfdir}/example/example.conf
}
 
FILES:${PN} += "${sysconfdir}/example/example.conf"

Details worth noticing

• ${D} prefixes the temporary destination; ${bindir} and ${sysconfdir} describe target paths. Omitting ${D} can write to the host or fail under task isolation.

• install -d creates directories with explicit intent; install -m sets predictable modes without preserving the build user’s ownership.

• FILES:${PN} controls which output package owns the configuration file; installation into ${D} alone does not guarantee packaging.

• The backslash continues one shell command inside the task. BitBake expands variables before the shell executes it.

7. do_package and package-write tasks

do_package analyzes ${D}, splits files according to PACKAGES and FILES:*, generates dependency metadata, and feeds QA/package-data work. Backend tasks such as do_package_write_rpm, do_package_write_deb, or do_package_write_ipk create repository artifacts according to PACKAGE_CLASSES.

• A recipe is not necessarily one binary package; inspect PACKAGES and the generated package split.

• do_package does not by itself mean an RPM, DEB, or IPK has been written—the selected do_package_write_* backend does that.

• Image construction later selects packages and builds a root filesystem; compiling a recipe does not automatically install it into every image.

• Packaging QA catches unshipped files, ownership, dependencies, debug paths, architecture, licenses, and other policy problems before deployment.

List tasks and build the normal target

$ bitbake -c listtasks recipename
bitbake recipename

What these commands establish

• Replace recipename with the recipe target, not an assumed binary package filename.

• listtasks shows tasks defined after metadata parsing; the set is more authoritative than a generic tutorial list.

• A plain bitbake recipename runs the default build target and its required dependency graph. This is the normal way to build a recipe.

• If output says tasks were unnecessary, stamps/signatures or shared state may already satisfy them; that is normal incremental behavior.

Run a task deliberately

$ bitbake -c fetch recipename
bitbake -c unpack recipename
bitbake -c compile recipename
bitbake -C compile recipename

Task-control semantics that prevent surprises

• -c compile requests do_compile and runs prerequisites that are not already satisfied; it is not equivalent to running only a compiler command.

• Task names are supplied without the do_ prefix on the CLI.

• -C compile recipename invalidates the compile stamp, then runs the recipe’s default target so downstream work is reconsidered.

• -f -c compile forces that task, but forcing one task does not automatically mean every downstream artifact is regenerated. Use the smallest action that matches the diagnosis.

Inspect variables, graphs and logs

$ bitbake -e recipename > recipename.env
bitbake -g recipename
bitbake -c devshell recipename
find tmp/work -path '*/recipename/*/temp/log.do_compile*' -print

How to use the evidence safely

• bitbake -e records the expanded recipe environment and variable history; it may expose private URLs or credentials, so redact before sharing.

• bitbake -g writes dependency graph files in the working directory; it does not build the target. Use Graphviz or targeted text searches to inspect them.

• devshell enters the configured recipe environment for interactive diagnosis. Changes made inside ${WORKDIR} remain disposable build artifacts.

• Task logs and run.do_* scripts under ${T} show what executed. Prefer the newest non-symlinked log when correlating a failure.

• The example find pattern may need the machine/tune-specific work path; bitbake -e recipename reveals ${T} directly.

Shared state changes what executes

BitBake calculates task signatures from relevant metadata and dependencies. When an eligible artifact with a matching signature exists in SSTATE_DIR or a configured mirror, a setscene task can restore it instead of executing the original task. Fast restoration is a correctness feature, not proof that BitBake ignored your recipe.

• A metadata change matters only if it affects the task signature; signature tools help explain unexpected reuse or rebuilds.

• do_clean removes recipe work output but preserves sstate, so rebuilding can restore results immediately.

• do_cleansstate also removes local sstate for the target but cannot erase remote mirror objects and is discouraged against shared caches during concurrent builds.

• Upstream documentation recommends forcing the required task rather than routinely using do_cleanall; do_cleanall also deletes downloads and can disrupt shared DL_DIR workflows.

Failure map

• Fetch failure: verify SRC_URI, revision, checksum, mirror policy, credentials, proxy, and network—not compiler flags.

• Unpack/patch failure: inspect archive layout, ${S}, patch order/context, selected append files, and source revision.

• Configure cannot find a library: inspect DEPENDS, recipe sysroot, PACKAGECONFIG, configure arguments, and cross-compilation tests.

• Compile fails only under BitBake: compare ${T}/run.do_compile*, ${T}/log.do_compile*, environment, parallelism, and undeclared host dependencies.

• Installed file missing from package: inspect ${D}, PACKAGES, FILES:*, package-split directories, and installed-vs-shipped QA.

• Package exists but image lacks it: inspect image/packagegroup install metadata, runtime dependencies, exclusions, compatibility, and rootfs logs.

• Task did not rerun: inspect stamps, signatures, overrides, and sstate before deleting caches.

A practical debugging sequence

1. Record the Yocto release/branch, layers, machine, distro, recipe provider/version, and exact command.

1. Use bitbake -c listtasks and bitbake -e to establish the resolved metadata rather than guessing.

1. Locate the first failed task and read its complete log plus generated run script.

1. Inspect that task’s inputs, working directory, dependencies, and expanded variables.

1. Reproduce with the smallest justified task command; avoid broad cache deletion that destroys evidence.

1. Run the normal recipe/image target afterward so packaging, QA, rootfs, and downstream dependencies are verified.

1. Convert the diagnosis into recipe, append, class, or configuration metadata and add CI coverage where appropriate.

Build record to keep with a bug report

• Exact command, first failed task, and unedited error context.

• Yocto/OE release and commit IDs for every relevant layer.

• MACHINE, DISTRO, provider/version choice, and significant configuration overrides.

• Task log, generated run script, and the relevant variable history from bitbake -e.

• Whether the result came from execution or sstate, plus the minimal reproduction and known-good comparison.

Related Yocto guides

• Resolve build/runtime dependency QA with fixing “rdepends but isn’t a build dependency”.

• Understand metadata composition with BitBake recipes and append files.

• Investigate package contents using Yocto package data and oe-pkgdata-util.

Primary references

• The Yocto Project task reference defines normal and manually invoked OpenEmbedded tasks.

• The BitBake command manual documents task, graph, environment, force, and execution controls.

• The Yocto Project build-directory reference documents downloads, shared state, work output, package data, and recipe sysroots.