CopyTarget
The CopyTarget tool copies files reliably from a source location to the destination location while maintaining file metadata, which includes ownership and permission.
CopyTarget supports the following operations:
• Copy file
• Create directory
• Update file metadata
• Update directory metadata
• Remove file
• Remove empty directory
• Create symlink
Prerequisites
• Ubuntu 18.04 x86_64 host
• python3 (>= 3.5)
• python3-yaml
• coreutils
• bash
Installing CopyTarget
The CopyTarget cool is packaged and distributed in the Debian software package (.deb) file. The following instructions describe how to download and install CopyTarget.
1. Download nvidia-driveos-<DRIVE_OS_release>-copytarget.deb to your workspace.
2. Install the Debian:
sudo apt install ./nvidia-driveos-<DRIVE_OS_release>-copytarget.deb
CopyTarget is installed on the host at /opt/nvidia/driveos/<DRIVEOS_RELEASE>/filesystems/CopyTarget/.
Using CopyTarget
Execute CopyTarget:
/opt/nvidia/driveos/<DRIVE_OS_release>/filesystems/copytarget/copytarget.py ${targetdir} MANIFEST.yaml
Where:
• <DRIVE_OS_release> is the release version of NVIDIA DRIVE OS.
• ${targetdir} is the destination directory.
• MANIFEST.yaml is the manifest file that specifies the file operations to execute.
Command Line Syntax
CopyTarget has the following invocation syntax with the following mandatory and optional arguments. For more information, see Command Argument Options.
copytarget.py ${targetdir} MANIFEST.yaml [MANIFEST1.yaml...] [OPTIONS]
The destination ${targetdir} and an initial manifest must be specified. Optional and additional manifests can be specified and are treated as if the initial manifest imported these additional manifests in its import section.
See Command Argument Options for additional options that can be specified to CopyTarget as part of [OPTIONS].
Command Argument Options
Argument | Description | Required | Default |
${targetdir} | Destination directory where CopyTarget conducts file operations at. | Required | - |
MANIFEST.yaml | Initial manifest required for CopyTarget to operate on. | Required | - |
--version, -v | Prints the version of CopyTarget. If this argument is supplied, CopyTarget does not continue execution after printing the version. | Optional | Not set. |
--help, -h | Prints the help message of CopyTarget. If this argument is supplied, CopyTarget does not continue execution after printing the help message. | Optional | Not set. |
CopyTarget Manifest
CopyTarget operates on one or more manifest files and at least one must be provided to the tool. Each manifest file contains the source and destination locations of the file, as well as the metadata. Manifests follow YAML syntax, a human-readable data-serialization language.
A manifest is specified by two sections: a header, and a file list. The header and file list serve different roles. In the next section, the roles of these sections are explained in detail.
Header
The header contains information required to process the file list section of the manifest. The header also provides information indicating the compatible CopyTarget software. The header includes the following information as captured and detailed in the table.
Attribute | Description | Required | Default | Domain | Example |
version | Defines the list of CopyTarget versions that the manifest is compatible with. | Mandatory | - | Decimal number | version: '1.4' |
exports | Subsection containing a list of key value pairs. The keys defined in this section are used as variable identifiers and using these keys in the filelist expands them into assigned values. | Optional | Empty list | Key: a valid variable name
Value: any valid YAML assignable value | exports:
- Key1: Value1
- Key2: Value2 |
imports | Subsection containing a list of additional manifests. | Optional | Empty list | List of zero, one, or more valid and compatible filepaths to additional CopyTarget manifest files. Filepaths can be absolute or relative to the current manifest file; and this relativity is applied recursively for each nested manifest file. Imported manifest files are processed first before continued processing of the current manifest; however, exports from the current manifest are processed and forwarded to nested manifests. Defining an export in the current manifest is made visible to the child and grandchild manifests. | imports:
- manifest1.yaml |
Filelist
The filelist section defines an itemized list of files to be copied from source to destination. The list is processed in the order specified in the section. The filelist contains zero, one, or more file sections. Each file section includes the following information as captured and detailed in the table.
Attribute | Description | Required | Default | Domain | Example(s) |
destination | Specifies the path of the file, directory, or symlink that needs to be copied to or created in the target directory. The name of the destination file does not need to match the filename specified in the source attribute; when there is a difference, the file will be renamed accordingly. | Mandatory | - | Valid Linux filepath. | destination: '${targetdir}/lib/lib-nv/lib_nvidia.so' |
source | Specifies the source path of a file or symlink that needs to be copied from. This attribute should not be set when creating a directory or changing the permission of an already existing file or directory. | Optional | "" | Valid Linux filepath. | source: '/drive-t186ref-linux/lib-targetfs/lib_nvidia.so' |
perm | Specifies the permission of the file or directory in octal notation. | Mandatory
(exception: creating symlinks) | - | Valid Octal notation of Linux permission. | perm: 0777 |
owner | Specifies the owner identifier of the file, directory, or symlink and can be assigned a numeric UID or a string specifying the username. | Mandatory | - | Valid Linux UID, or UID alias. | owner: root
owner: 0 |
group | Specifies the group identifier of the file, directory, or symlink and can be assigned a numeric GID or a string specifying the groupname. | Mandatory | - | Valid Linux GID, or GID alias. | group: root
group: 0 |
create_symlink | If set to true, a symlink is created at the destination location. The symlink points to the value specified in the attribute source. By default, this field is set to false. | Optional | false | true, false | create_symlink: true |
remove | If set to true, the file specified in the destination attribute is removed. If the destination attribute points to a directory, the directory must be empty; otherwise, CopyTarget reports an error; this is intentional and prevents inappropriate deletion. By default, this field is set to false. | Optional | false | true, false | remove: true |
CopyTarget File Operations
CopyTarget supports a limited number of file operations and to have CopyTarget execute a particular operation, a combination of file attributes must be specified. Any invalid combination is raised as an error by CopyTarget and an error message with a pointer to the offending file attribute is presented to the user to resolve.
The following table describes each of the valid operations and the required attributes to trigger the operation.
File Operation | Required Attributes | Example(s) |
Copy | Attribute | Requirement | destination | Required. | source | Required (unless an imported manifest already defines it). | perm | Required (unless an imported manifest already defines it). | owner | Required (unless an imported manifest already defines it). | group | Required (unless an imported manifest already defines it). | create_symlink | Can be absent (default is false) or must be set to false. | remove | Can be absent (default is false) or must be set to false. |
| Copy file from ${}/libraries/lib_nvidia.so to ${targetdir}/usr/lib/lib_nvidia.so with ownership root:root and permission 0777; where ${targetdir} is the target directory provided to CopyTarget. Attribute | Value | destination | ${targetdir}/usr/lib/lib_nvidia.so | source | ${WORKSPACE}/libraries/lib_nvidia.so | perm | 0777 | owner | root | group | root | create_symlink | false | remove | false |
|
Create Directory | Attribute | Requirement | destination | Required (must end with a trailing slash; otherwise, an error is reported). | source | Must be absent. | perm | Required (unless an imported manifest already defines it). | owner | Required (unless an imported manifest already defines it). | group | Required (unless an imported manifest already defines it). | create_symlink | Can be absent (default is false) or must be set to false. | remove | Can be absent (default is false) or must be set to false. |
| Create directory at ${targetdir}/usr/lib/nvidia/ with ownership root:root and permission 0777; where ${targetdir} is the target directory provided to CopyTarget. Attribute | Value | destination | ${targetdir}/usr/lib/nvidia/ | source | | perm | 0777 | owner | root | group | root | create_symlink | false | remove | false |
|
Update File Metadata | Attribute | Requirement | destination | Required. | source | Must be absent. | perm | Required (unless an imported manifest already defines it). | owner | Required (unless an imported manifest already defines it). | group | Required (unless an imported manifest already defines it). | create_symlink | Can be absent (default is false) or must be set to false. | remove | Can be absent (default is false) or must be set to false. |
| Set file ${targetdir}/usr/lib/lib_nvidia.so with ownership nvidia:nvidia and permission 0644; where ${targetdir} is the target directory provided to CopyTarget. Attribute | Value | destination | ${targetdir}/usr/lib/lib_nvidia.so | source | | perm | 0644 | owner | nvidia | group | nvidia | create_symlink | false | remove | false |
|
Update Directory Metadata | Attribute | Requirement | destination | Required. | source | Must be absent. | perm | Required (unless an imported manifest already defines it). | owner | Required (unless an imported manifest already defines it). | group | Required (unless an imported manifest already defines it). | create_symlink | Can be absent (default is false) or must be set to false. | remove | Can be absent (default is false) or must be set to false. |
| Set directory ${targetdir}/usr/lib/nvidia/ with ownership nvidia:nvidia and permission 0644; where ${targetdir} is the target directory provided to CopyTarget.
Note: This operation is not recursive and only updates the metadata of ${targetdir}/usr/lib/nvidia/. Attribute | Value | destination | ${targetdir}/usr/lib/nvidia/ | source | | perm | 0644 | owner | nvidia | group | nvidia | create_symlink | false | remove | false |
|
Remove a File | Attribute | Requirement | destination | Required. | source | Should be absent. | perm | Should be absent (if specified, is ignored). | owner | Should be absent (if specified, is ignored). | group | Should be absent (if specified, is ignored). | create_symlink | Should be absent (default is false) or must be set to false. Setting this to true causes an error to be thrown. | remove | Must be set to true. (unless an imported manifest already defines it). |
| Remove file ${targetdir}/usr/lib/lib_nvidia.so; where ${targetdir} is the target directory provided to CopyTarget. Attribute | Value | destination | ${targetdir}/usr/lib/lib_nvidia.so | source | - | perm | - | owner | - | group | - | create_symlink | false | remove | true |
|
Remove Empty Directory | Attribute | Requirement | destination | Required. | source | Should be absent. | perm | Should be absent (if specified, is ignored). | owner | Should be absent (if specified, is ignored). | group | Should be absent (if specified, is ignored). | create_symlink | Must be set to false (unless an imported manifest already defines it). | remove | Must be set to true (unless an imported manifest already defines it). |
| Remove empty directory ${targetdir}/usr/lib/nvidia/; where ${targetdir} is the target directory provided to CopyTarget. Attribute | Value | destination | ${targetdir}/usr/lib/nvidia/ | source | - | perm | - | owner | - | group | - | create_symlink | false | remove | true |
|
Create Symlink | Attribute | Requirement | destination | Required. | source | Required (unless an imported manifest already defines it). | perm | Should be absent (if specified, is ignored). | owner | Required (unless an imported manifest already defines it). | group | Required (unless an imported manifest already defines it). | create_symlink | Must be set to true (unless an imported manifest already defines it). | remove | Must be set to false (unless an imported manifest already defines it). |
| Create symlink at ${targetdir}/usr/lib/lib_nvidia.so.1 pointing to /usr/lib/lib_nvidia.so; where ${targetdir} is the target directory provided to CopyTarget. Attribute | Value | destination | ${targetdir}/usr/lib/lib_nvidia.so.1 | source | /usr/lib/lib_nvidia.so | perm | - | owner | nvidia | group | nvidia | create_symlink | true | remove | false |
|
Importing Manifests
Zero, one, or more additional manifests can be specified in the parent manifest. Further nested manifests can also be recursively achieved; since child manifests can have additional manifest imports. This behavior allows for hierarchical overlaying.
Overriding Parent Attributes
Hierarchical manifests allow for child manifests to be overridden, additional attributes to parent to be added, or ancestor manifests.
For example, a given child manifest may define a file attribute to have a permission attribute of 0777 and a parent manifest can override this permission attribute at will with another value such as 0666.
Mapping Overlays using Destination Field
Overlaying of manifests require the use of the destination key. When the destination key matches that of one of the parent manifest's specified file attribute, any values specified in the particular child manifest overlays (overrides) the running set of updated values, which in turn can be updated by another intermediate parent manifest.
Therefore, for overlaying to operate effectively, destination keys must match. Any destination keys that do not match are treated as a new file attribute, with its own set of specified required attributes. One or more missing attributes result in an error.
Sequence of Manifest Overlaying
The sequence of manifest processing is dictated by the ordering specified in the import section of the parent manifest and the processing of this list follows a depth first list.
Consider the following set of manifest examples, which causes the final file attribute to have a resultant value depending on the ordering of the root most manifest (which is provided first to CopyTarget). See the result below.
#manifest0.yaml
imports:
- manifest1a.yaml
- manifest1b.yaml
fileList:
- destination: /destination.txt
source: /source.txt
perm: 644
owner: 0
group: 0
#manifest1a.yaml
imports:
- manifest2a.yaml
- manifest2b.yaml
fileList:
- destination: /destination.txt
source: /source.txt
perm: 644
owner: 1000
group: 1000
#manifest2a.yaml
fileList:
- destination: /destination.txt
source: /source.txt
perm: 655
owner: 2000
group: 1500
#manifest2b.yaml
fileList:
- destination: /destination.txt
source: /source2.txt
perm: 777
group: 3000
#manifest1b.yaml
fileList:
- destination: /destination2.txt
source: /source.txt
perm: 644
owner: 1000
group: 1000
Resultant Table
Starting Manifest | Copied File(s) | Ownership | Permission |
manifest0.yaml | 1. /source2.txt to /destination.txt 2. /source.txt to /destination2.txt | 1. 0:0 2. 1000:1000 | 1. 0644 2. 0644 |
manifest1a.yaml | 1. /source2.txt to /destination.txt | 1. 1000:1000 | 1. 0644 |
manifest1b.yaml | 1. /source.txt to /destination2.txt | 1. 1000:1000 | 1. 0644 |
manifest2a.yaml | 1. /source.txt to /destination.txt | 1. 2000:1500 | 1. 0655 |
manifest2b.yaml | Error, because owner attribute is not defined. | - | - |
Exports
The exports section can be used for variable dereferencing in the current and child manifests.
Syntax
The syntax follows a key value pair, delimited by a colon. This is follows the standard YAML syntax format. More than one export can be specified. The key must not have any spaces.
For example, the following code statement assigns the value VALUE to the variable named KEY.
exports:
- KEY: VALUE
Dereferencing a Variable
To dereference or otherwise obtain the value stored in a particular key of an export, enclose the name of the variable between opening and closing braces {} and prepended with a dollar sign $.
For example, the variable named KEY can be dereferenced when specified as ${KEY}, as illustrated in the following example, which sets the attribute destination to VALUE.
exports:
- KEY: VALUE
fileList:
- destination: ${KEY}
Overriding Parent Assigned Values
Values assigned previously to a variable from a parent manifest can be overriden by any child manifest and thus taken forward.
Variable assignments are part of the preprocessing step, which means that all variables are assigned the final values prior to iteration of any filelist section. This is intended to prevent variable value aliasing problems.
For example, in the following scenario, the attribute source of destination dest is assigned the value FINAL3, as would the destination dest2. The source attribute of destination dest2 is never assigned the value FINAL2, even though it appears in the intermediate manifest import because of the nature of preprocessing requirement.
#manifest1c.yaml
exports:
- KEY: FINAL1
imports:
- manifest2c.yaml
- manifest2d.yaml
fileList:
- destination: dest
source: ${KEY}
#manifest2c.yaml
exports:
- KEY: FINAL2
fileList:
- destination: dest
source: ${KEY}
- destination: dest2
source: ${KEY}
#manifest2d.yaml
exports:
- KEY: FINAL3
fileList:
- destination: dest
source: ${KEY}
Examples: Creating CopyTarget Manifest
Copying a File
The following file attribute instructs CopyTarget to copy the file from ${WORKSPACE}/home/nvidia/file.txt to ${targetdir}/file.txt with permission 0644 and ownership nvidia:nvidia, where ${targetdir} is the target directory provided to CopyTarget.
fileList:
- destination: ${targetdir}/file.txt
source: ${WORKSPACE}/home/nvidia/file.txt
perm: 644
owner: nvidia
group: nvidia
Creating a Directory
The following file attribute instructs CopyTarget to create the directory ${targetdir}/usr/bin/ with permission 0644 and ownership nvidia:nvidia, where ${targetdir} is the target directory provided to CopyTarget.
Note: | If the destination directory already exists, the metadata of the ${targetdir}/usr/bin/ is set to values specified, overriding any original values.. |
fileList:
- destination: ${targetdir}/usr/bin/
perm: 644
owner: nvidia
group: nvidia
Updating a File's Metadata
The following file attribute instructs CopyTarget to set the file ${targetdir}/file.txt with permission 0644 and ownership nvidia:nvidia, where ${targetdir} is the target directory provided to CopyTarget.
fileList:
- destination: ${targetdir}/file.txt
perm: 644
owner: nvidia
group: nvidia
Removing a File
The following file attribute instructs CopyTarget to remove the file ${targetdir}/file.txt, where ${targetdir} is the target directory provided to CopyTarget.
fileList:
- destination: ${targetdir}/file.txt
remove: true
Creating a Symlink
The following file attribute instructs CopyTarget to create a symlink at ${targetdir}/link.txt that points to /file.txt, where ${targetdir} is the target directory provided to CopyTarget.
fileList:
- destination: ${targetdir}/link.txt
source: /file.txt
create_symlink: true
Comprehensive Example: copytarget-manifest.yaml
An example of a comprehensive copytarget-manifest.yaml looks like the following.
version: '1.4'
# Exports environment variables
exports:
- SAMPLES_HOME: /home/nvidia/drive-t186ref-linux/samples/
- FILE_VERSION: 1
# Itemised file list
fileList:
# The following attribute creates a directory "${targetdir}/nvidia/"
# with ownership 0:0 and permission 0755; where ${targetdir} is the target directory specified to CopyTarget.
- destination: ${targetdir}/nvidia/
perm: 0755
owner: 0
group: 0
# The following attribute copies the file from "${WORKSPACE}/home/drive/samples/nvidia_sample"
# to "${targetdir}/nvidia/nvidia_sample" with ownership 1000:1000 and permission 0777
- destination: ${targetdir}/nvidia/nvidia_sample
source: ${WORKSPACE}/home/drive /samples/nvidia_sample
perm: 0777
owner: 1000
group: 1000
# The following attribute creates a symlink at "${targetdir}/nvidia_sample_1"
# pointing to "/nvidia/nvidia_sample"; where ${targetdir} is the target directory specified to CopyTarget.
- destination: ${targetdir}/nvidia_sample_${FILE_VERSION}
source: /nvidia/nvidia_sample
create_symlink: true
# The following attribute removes file "${targetdir}/nvidia/nvidia_sample";
# where ${targetdir} is the target directory specified to CopyTarget.
- destination: ${targetdir}/nvidia/nvidia_sample
remove: true
Errors
To enable proper tracking of files and to prevent erroneous file copies, CopyTargetv1.4 enforces the following guidelines. Violations are reported with an error and the line of the offending code.
Fault | Description | Error |
Files cannot be copied implicitly. | Copying directories is not permitted. | In case of violation, CopyTarget throws an error saying, for example, Error: CopyTarget does not allow directory to directory copy. Please itemize the files to copy. |
Use of wildcards in file names in not permitted. | Wildcard expansion is not permitted. | In case of violation, CopyTarget throws an error saying, for example, FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/*'. |
Directory path entries must end with a slash (/). | Directories must end in a trailing slash to differentiate from files. | In case of violation, CopyTarget throws an error saying Error: Directory entries must end with a trailing slash. |
Metadata of each file must be defined. | All metadata must be specified. | In case of violation, CopyTarget throws an error saying, for example, Error: Expected key 'owner' is not defined. |
