Skip to content

External Dependencies

Overview

External dependencies are a way within the build environment to describe external dependencies and have Stuart fetch them when doing the stuart_update operation. Stuart will also verify the ext_dep state when doing a build to ensure the environment is in the required state prior to building.

Ext_deps have solved three major issues.

  1. Binaries causing bloat of git repositories
  2. Conditional inclusion of a dependency (only for certain usages)
  3. Reproducibility and tracking of dependencies

Why

Git Bloat

Best practices advise against checking in binaries to git repositories as the overall size of git repos will balloon quickly causing slow clones and slow operations. Building firmware often requires custom tools, firmware blobs, or other binaries and it is critical these are maintained and versioned with the repository. Package management tools can solve the hosting of these binaries but edk2 has no built in tool to track them, extract them, etc. Ext_deps provide that mechanism.

Conditional Inclusion (scopes)

Ext_deps leverage the environment scope concept so that a repository can carry ext_deps that are only used in some conditions. Scopes are a string that an invocable tool uses to indicate what ext_deps should be used. These scopes are loosely based on functionality.

Reproducibility and Tracking

Ext_deps are common infrastructure so that all external dependencies can be handled consistently. Versions are added to the version report so that for any given operation (like build) a complete list of what was used is available. This makes tracking versions consistent and "free". Ext_deps when fetched will update their state. If the repository is updated to include a new ext_dep version the tool will be told the environment state is not valid and can then enforce that the user updates their environment.

Examples of Usage

Here are a few examples where ext_deps have been found useful:

  • An ext_dep describing a test repository that is only needed when running unit tests. By leveraging scopes this ext_dep is only fetched when the unittest scope is active.
  • Similar to the unit test dependency, support for CI builds often require unique dependencies. When doing a CI build of a core repository it might have critical dependencies that need to be fetched but when the core repository is included within a platform repository as a dependency, then the core would defer to the platform as to how to include the dependency.
  • An ext_dep describing the compiler toolchain. This ext_dep is only needed when a builder is using that toolchain for that target type.
  • An ext_dep describing some platform binary. This is only needed when building that given platform and since git is not optimized to handle binaries this saves a lot of unnecessary bloat in the repository.

Supported Types

NuGet Dependency

Nuget dependency is used to fetch files from a nuget feed. This feed can be either unauthenticated or authenticated. Support is done by using the nuget command line tool. When the ext_dep type is set to nuget the descriptor will be interpreted as a nuget dependency. Nuget has a few nice features such as caching, authentication, versioning, and is platform and language agnostic. This package has a version of the NuGet.exe binary or a user can configure their environment to use a defined version by setting NUGET_PATH to the folder containing the NuGet.exe that should be used.

Web Dependency

Web dependency is used to describe a dependency on an asset that can be downloaded via a URL and a web request. It will download whatever is located at the source URL and can support single files, compressed files, and folders. When the ext_dep type is set to web the ext_dep will be interpreted as a web dependency.

Git Dependency

Git dependency is used to describe a dependency on a git repository. This repository will be cloned to the ext_dep location and the version will be checked out. For this ext_dep descriptor the type is git. A git dependency should be treated as read-only because the verify and clean phase will do destructive operations where local changes would be destroyed.

Azure Universal Packages Dependency (az-universal)

Azure Universal packages are yet another package management solution. These packages are supported by Azure DevOps server and use the cross platform Azure CLI tool to retrieve the content. To use this ext_dep type the development environment must have the Azure CLI tool available and the devops extension. Since anonymous feeds are not supported your environment must already be authenticated or you must define a system environment variable that will hold a DevOps PAT with package read authorization.

More information:

Developer Note

To create a new Dependency type it requires a new subclass of the ExternalDependency class. The subclass needs to have a type field and then factory method in ExternalDependency.py needs to be updated to create new instances of the new type.

Local Caching

In some cases (e.g. a CI server), it is necessary to set up and tear down multiple repos and/or branches, and this can severely impact the network time to fetch all new dependencies. While Nuget has it's own caching system, this does not scale to the other dependency types.

To address this, Stuart provides the STUART_EXTDEP_CACHE_PATH environment variable. If this variable is set, and points to a valid directory on the local filesystem, it will be populated with a copy of any extdep that is fetched by Stuart during the process of a build. This cache will be checked prior to invoking any type-specific extdep fetch operation. If the extdep already exists in the cache, it will be copied from the local cache into its final destination.

This cache will keep unique entries for versions and sources. In other words, multiple versions of the same extdep from the same source will be cached independently, but the same version from multiple sources will also be cached independently. This is to prevent possible cache corruption when consuming external code that may have a name collision.

A note on NuGet on Linux

If you're using Linux (whether through WSL or natively), you'll need to add the proper sources to your relevant package manager for mono. As of time of writing, the Ubuntu mono packages are out of date.

You can follow the instructions here: https://www.mono-project.com/download/stable/#download-lin

Here are the instructions (as of time of writing):

sudo apt install gnupg ca-certificates
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb https://download.mono-project.com/repo/ubuntu stable-bionic main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
sudo apt update

sudo apt upgrade (if mono is already installed)
- or -
sudo apt install mono-devel

If you're running a different kind of package manager, or on a system without a package manager, visit the link above for instructions on your platform. If you're on a system that NuGet natively supports (currently only Windows) you won't need to install Mono.

How external dependencies work

Ext_deps are found by the SDE (self-describing environment). If you have any questions about that, go review the document for that. Once the ext_dep is found it can be interacted with depending on use case/tool. Objects created with the data from ext_dep descriptors and are subclassed according to the "type" field in the descriptor.

These objects contain the code for fetching, validating, updating, and cleaning dependency objects and metadata. When referenced from the SDE itself, they can also update paths and other build/shell vars in the build environment.

How to create/use an ext_dep

An ext_dep is defined by a json file that ends in_ext_dep.json It must follow the schema outlined below. It will be unpacked in a new folder in the same directory as the .json file in a folder named {name}_extdep.

We strongly recommend adding any folder that ends in _extdep to your repositories .gitignore. It would look like this:

*_extdep/

Ext_Dep Example json file

{
"scope": "cibuild",
"type": "nuget",
"name": "iasl",
"source": "https://api.nuget.org/v3/index.json",
"version": "20190215.0.0",
"flags": ["set_path", "host_specific"]
}

If you wish to create a new ext_dep type other then those mentioned in Supported Types, See Creating an Extdep Type

The base schema

Required

scope: (required) (string) - name of scope when this ext_dep should be evaluated

  • type: (required) (string from list of known types) - See above for types
  • name: This is the name of the ext_dep and will be part of the path where the ext_dep is unpacked
  • source: see per type
  • version: see per type
  • flags: Optional conditions that can be applied. Can be empty list

Optional

  • id: (string) - Identifier allowing override feature - Must be unique
  • override_id: (string) - Identifier of the ext_dep this should replace (allows for changing an ext_dep in another source by id)
  • var_name: TODO
  • error_msg: an optional message that will disabled to the user if this ext_dep fails to fetch

Nuget Type Schema differences

  • source: This should be the nuget feed URL
  • version: nuget version. Generally xx.yy.zz

For this type there are zero additional ext_dep fields.

Web Type Schema differences

  • source: url to download
  • version: only used for folder naming

For this type there are three additional ext_dep fields:

internal_path (required)

This describes the internal structure of whatever we are downloading.

If you are just downloading a file, include the name you would like the file to be.

If you are downloading a directory, indicate so with a / before the path. The folder the path points to will have it's contents copied into the final name_ext_dep folder.

compression_type (optional)

Including this field is indicating that the file being downloaded is compressed and that you would like the contents of internal_path to be extracted. If you have a compressed file and would not like it to be decompressed, omit this field.

Currently tar and zip files are supported. If the file is not compressed, omit this field.

sha256 (optional)

If desired, you can provide the hash of your file. This hash will be checked against what is being downloaded to ensure it is valid. It is strongly recommended to use this to ensure the contents are as expected.

Git Type Schema Differences

  • source: url of git repo
  • version: commit hash to checkout (short* or long)

* Short is considered the first 7 digits of a commit hash

Experimental Option: url_creds_var

If this field is found in the descriptor file when initializing this extdep, the string value listed will be checked against the environment's shell_vars. If a matching var is found, this string in the shell_var will be prepended to the URL host for the source URL.

NOTE: This is intended for server builds and may be subject to change as we figure out how it fits into build flows. Also note that any credentials passed may end up in build logs and other server-side artifacts. Use with caution!

Example:

{
    "scope": "global",
    "type": "git",
    "name": "ExampleRepo",
    "source": "http://example.com/path/to/repo.git",
    "version": "7fd1a60b01f91b314f59955a4e4d4e80d8edf11d",
    "url_creds_var": "my_url_creds",
    "flags": []
}

# Populate shell var.
env.set_shell_var('my_url_creds', 'my_user:my_pass')

# URL cloned by the GitDependency object will look like...
final_url = 'http://my_user:my_pass@example.com/path/to/repo.git'

Azure Universal Packages Type Schema Differences

For this type there are three additional ext_dep fields:

feed

Name of the Azure DevOps feed

project (optional)

Name of the project for project scoped feeds. If not defined it will be treated as an organizational scoped feed.

name

Name of the artifact

file-filter (optional)

Filter for folders and files to download. Azure Universal packages supports only partial downloads. See https://docs.microsoft.com/en-us/azure/devops/artifacts/quickstarts/universal-packages?view=azure-devops#download-specific-files for more details.

pat_var (optional but suggested)

OS shell variable that contains a personal access token (PAT) for this ext_dep. Universal packages do not support anonymous access therefore the developer needs to authenticate using az login or define a PAT. The PAT needs to have Packaging - Read access.

Example:

{
  "scope": "global",
  "type": "az-universal",
  "name": "hello-world-file",
  "source": "https://dev.azure.com/tianocore",
  "project": "edk2-pytool-extensions",
  "version": "1.0.0",
  "feed": "ext_dep_unit_test_feed",
  "pat_var": "PAT_FOR_UNIVERSAL_ORG_TIANOCORE",
  "flags": []
}

The Flags

There are specific flags that do different things. Flags are defined by the SDE and cannot be modified without updating the pip module. More information on the flags can be found in the SDE documentation.