BYOB Configuration

Unfortunately, it is not easy to resolve an s3 URI to an ICAv2 (or vice versa), due to API limitations on storage configurations.

Hence you will need three separate environment variables that point to three separate yaml files :) This will include:

A storage configuration setup
- What is the bucket / prefix for each storage configuration (saves an API call each time we need to convert an S3 URI to an ICAv2 URI
A project to storage configuration mapping
- which project is configured to which storage configuration, and at what prefix (if any).
A storage credential setup
- For a given storage credential id (that the user either created or has been shared across the tenant), what are the s3 prefixes that this storage configuration has access to?

Storage Configuration Setup

Env var:

ICAV2_STORAGE_CONFIGURATION_LIST_FILE=/path/to/storage-configuration-file.yaml

Yaml file contents:

List of objects where each object has the following attributes:

id: The storage configuration id
bucketName: The name of the bucket
keyPrefix: The s3 prefix for this storage configuration object
storageCredentialId: The storage credential id used to build this storage configuration

Your storage configuration yaml file may look like this:

- id: 81657569-adce-4ae6-bd0d-87225fe819e9
  bucketName: reference-data-bucket
  keyPrefix: reference-data
- id: af421f82-4127-4cee-94e7-f249ff4ddc43
  bucketName: research-project-bucket
  keyPrefix: research-data/

You may be able to generate the storage configuration mapping programatically with the following command:

curl --fail --silent --location \
  --request "GET" \
  --url "https://ica.illumina.com/ica/rest/api/storageConfigurations" \
  --header "Accept: application/vnd.illumina.v3+json" \
  --header "Authorization: Bearer ${ICAV2_ACCESS_TOKEN}" | \
jq --raw-output \
  '
    .items |
    map(
      {
        "id": .id,
        "bucketName": .storageConfigurationDetails.awsS3.bucketName,
        "keyPrefix": .storageConfigurationDetails.awsS3.keyPrefix
      }
    )
  ' | \
yq --prettyPrint --unwrapScalar

Project to Storage Configuration List

Env var:

ICAV2_PROJECT_TO_STORAGE_CONFIGURATION_MAPPING_LIST_FILE=/path/to/project-to-storage-configuration-file.yaml

Yaml file contents:

List of objects where each object has the following attributes:

id: The id of the project
name: The name of the project
storageConfigurationId: The storage configuration that the project is mounted on
prefix: The prefix of the project on the storage configuration
- if the project is set to the root of the storage configuration, set the prefix parameter to null

Your project-to-storage configuration yaml may look something like this

- id: 81657569-adce-4ae6-bd0d-87225fe819e9
  bucketName: reference-data-bucket
  storageConfigurationId: 81657569-adce-4ae6-bd0d-87225fe819e9
  # Project mounted at the root of the prefix (s3://research-data-bucket/reference-data/)
  prefix: null
- id: 33103584-2531-4d10-8d2e-fd3fa16aeb7c
  name: colon-cancer-project
  storageConfigurationId: af421f82-4127-4cee-94e7-f249ff4ddc43
  # Project mounted at the prefix (s3://research-project-bucket/research-data/colon-cancer-data/)
  prefix: colon-cancer-data

Storage Credential Setup

Env var:

ICAV2_STORAGE_CREDENTIAL_LIST_FILE=/path/to/storage-credentials.yaml

Yaml file contents:

List of objects where each object has the following attributes:

id: The id of the storage credentials
name: The name of storage credentials
s3UriList: The list of objects with the following keys:
- bucketName: A bucket that this storage credential has access to
- keyPrefix: A key prefix on the bucket that this storage credential has access to

Your storage credential yaml may look something like this

# Storage credentials for ICAv2
# Reference data
- id: e737fcdd-d61c-4bd7-9a7b-ff70fe88d405
  name: icav2_ref_data_aws_user
  s3UriList:
    - bucketName: reference-data-bucket
      keyPrefix: refdata/
# Research data
- id: 33103584-2531-4d10-8d2e-fd3fa16aeb7c
  name: icav2_research_data_aws_user
  s3UriList:
    - bucketName: research-project-bucket
      keyPrefix: research-data/colon-cancer-data/
    - bucketName: research-project-bucket
      keyPrefix: clinical-trial-control-data/colon-control-data/