ADR-26-03: Selection of Storage Solution for Pre-Clinical Imaging Platform

Status

Accepted

Impact

Medium

Demand Owner

Jon Fuller, Aaron Archer Waterman, Lindsay Liang

ADR Contributors

Anish Kejariwal

Reviewers

Architecture Council

Informed

Architecture Council, Heads of CSCoE-DDC, Marius Quadflieg, Paul Saraceno, Carlos Fenoy, Moritz Gilsdorf, Fabia Fricke, James Fitzpatrick

Consulted stakeholders

Imran Khan, Anish Kejariwal, Alex Weigel, Christian Blumenroehr, Stefan Fraessle, Malvin Leuenberger, Ben Freiberg

Publication Date

May 7th, 2026

📗 Background

The existing storage landscape was extensively evaluated as part of the preparation of the implementation plan, including the landscaping work by McKinsey. Currently, the landscape involves bare RCP accounts (S3, EFS, etc), Data Ecosystem, Minerva (includes Nebula), Arvados, Big DipIR, ROSS, sHPC gpfs, CIFS file shares, Endor. In addition to these storage platforms/options, some labs were provisioning their own local file storage solutions, and some data was simply stored on acquisition instruments. Further to the above, there are three archiving solutions Windream (which will be decommissioned in April 2026), and OneArchive, Arvados.

ℹ️ Context

We need to provide scalable (storage volume, and bandwidth), cost effective, well governed storage across multiple geographies for image data, and related analysis artifacts (e.g. object detection masks, cell counts). The storage needs to be well integrated with adjacent technologies, in particular data transfer, image registration and compute.

In this ADR we will only cover storage requirements as they pertain to ‘long term storage’. In cases where temporary caches of data are required, the relevant part of the solution stack will need to make these considerations (e.g. temporary cache as part of data ingestion process, temporary cache as part of a processing pipeline). Additionally, we note that the scope of data migrations to the PCI ecosystem is not yet completely explored. It’s likely that some portion of existing imaging data may be migrated to the PCI ecosystem. The issues of temporary caches, and data migrations can be evaluated in separate tech reviews.

The decision must account for the following drivers:

Business Goal: Centralize the management of PCI data to support good data governance, and adherence to legal requirements. Ability to support requirements for data access stemming from the RDE WS7 PBAM project, and additional requirements stemming from other regulations and guidelines (e.g. US DoJ Data Security Program)
Technical Constraint: Data volumes and velocities. Ability to support data residency and retention requirements (e.g. European and US data residency requirements as a minimum). Ability to support automated (?) tiered storage for effective cost control. Effective support for building event-driven architectures.
xRED Constraint: Processes need to scale to allow onboarding all in-scope labs and instruments during the ramp-up phase of the PCI program (this is currently x, and y instruments). The storage solution needs to be able to support the pRED/gRED SOPs for data retention, disaster recovery, and data ownership. Ability to support a variety of authentication use cases - including client, machine-to-machine (m2m), most likely via OAuth 2.0 authentication flows (PingFed and Janus).

🤝 Outcome / Decision

We have decided to proceed with Option A: Data Ecosystem S3 Buckets.

Justification

Data Ecosystem strikes the right balance between exposing well-known, underlying technology (S3) while providing a cohesive set of policies and tagging standards that improve data discoverability and enable cost management across groups and initiatives. Furthermore, it closely aligns with CS CoE strategy to build around platforms, ensuring that PCI is able to leverage economies of scale within the CoE.

Key factors in this decision:

Low cognitive overhead: As a lightweight wrapper around S3, there is minimal learning curve for engineers already familiar with AWS. No new domain-specific concepts or APIs to learn (unlike Arvados, which introduces its own data model and event abstraction).
Self-service provisioning: The Data Ecosystem team provides a self-service tool ("vending machine") for bucket provisioning. Currently self-service supported regions include:
- us-west-2 (Oregon)
- us-east-1 (N. Virginia)
- us-east-2 (Ohio)
- eu-central-1 (Frankfurt)
- eu-central-2 (Zurich)
- eu-west-2 (London)
- Additional regions can be supported, but are not currently self-service.
Flexible policy management: The self-service tool allows straightforward creation and editing of bucket policies, particularly for engineers already familiar with S3 bucket policy syntax.
Integration-ready: The vending machine includes quick-toggle options to grant access to key platform principals:
- BiFrost (data movement service)
- Darkroom (image registration service)
Event-driven architecture support: The DE Storage team has rolled out a feature to connect s3:PutObject events from specified bucket prefixes directly into Darkroom — a heavily desired capability. This also establishes the framework for enabling event bus connections to future event-based services.
Loosely coupled: Data Ecosystem is a standalone storage service, not tightly coupled to any specific workflow engine, catalog, or registration service. This preserves flexibility for downstream architectural decisions.
Integration with AWS IAM significantly simplifies integrations with other AWS / RCP hosted systems/platforms (e.g. Minerva, MLOps)

Solutions Considered

Option A: Data Ecosystem S3 Buckets

Collection of S3 buckets managed under a single RCP account by a dedicated engineering team. Provides a centralized, governed storage layer with consistent policies while remaining close to "raw" S3 capabilities.

Option B: RCP S3 Buckets

Direct S3 bucket provisioning within individual RCP accounts, with no centralized management layer. This is the most "raw" S3 approach — teams provision and manage their own buckets independently.

Option C: Minerva S3 Buckets

AWS S3 buckets provisioned and managed by the Minerva team.

Option D: Arvados

Open-source data management platform (GitHub) with storage component "Keep" and workflow orchestration via "Crunch". In the context of this storage tech review, we are only interested in the functionality of Keep.

Option E: Endor

Endor is positioned in the Science & Research domain as a data backbone for supporting digital labslab. Currently Endor is hosted on the Navify platform (using AWS technologies). It currently supports C3 data (C4 data support is planned). It uses AWS S3 as the primary storage layer, and enables automatic tiering, with data ownership and observability implemented.

Evaluation Process

We evaluated options based on the following weighted priorities:

Technical Suitability: Does it solve the core problem effectively?
Engineering Velocity (Time-to-Value): Can the feature team deliver this independently, or does it require external dependencies?
Self-Service Maturity: Does this solution allow engineers to "pave their own road" (e.g., via Terraform/CDK) or does it require manual ticket filing?
Cost: Is the solution cost effective, and does it allow for monitoring and minimizing costs.
Operational Complexity: What is the burden of maintenance on the team?
AWS Native Integration: How well does it leverage existing managed services (reducing ops overhead)?

Features

Criteria

Feature

Criticality

Compliance

The solution needs to allow storing data in a manner that can be aligned with at least the following regulatory, and security requirements. GDPR (and equivalent local regulations) Encryption of data at rest COREMAP - C3/C4 data Support for pRED disaster recovery SOP (and gRED equivalents). DoJ DSP directive German Workers Council RAIA

Must-have

Data residency

The system needs to allow storage of data in the US and Europe.

Must-have

Data lifecycle

The system needs to seamlessly transition data between hot and cold(er) storage tiers in order to reduce costs where appropriate.

Should-have

Archiving

The system needs to provide capabilities to allow archiving data compliant to the COREMAP requirements.

Must-have

Data volume/velocity

File sizes can vary from KBs to TBs. Larger ingestions are currently of the order 100,000 images (300GB). We anticipate ingestion into the system to be rate limited by the network connection between Roche sites and the cloud storage provider. We anticipate that analysis reading/writing from the storage may be massively parallel (scaling to 1000s parallel tasks). We must plan for 10x/100x data volume and velocity in the near future.

Open/unclear

Data access authorization

The storage solution needs to support authorization to access data stored within the system. Authorization needs to be determined subject to the data sharing rules defined by RDE WS7 PBAM, and additional criteria.

Should-have

Data ownership

The storage solution must be able to attribute data ownership to an individual - in order that they can be accountable for decisions around the data. This ownership must be possible to be transferred (e.g. in case an employee changes role). See Roche Group Records Management Directive

Must-have

Data types

The PCI platform will not be storing only image data formats (e.g. TIFF, JPG, NDPI, Zarr, etc.), but also raw data (e.g. Bruker NMR files), and additionally ‘image adjacent’ formats, (e.g. segmentation masks, cell detections, etc). The list of these formats will necessarily evolve over time in order to support evolving needs of researchers.

Should-have

Unique & Persistent Identifiers (GUPRI)

A GUPRI can safeguard data links over time. Catalogued data should be associated with robust, globally unique, and persistent digital IDs (GUPRI) that do not break or change when data is logically reorganized, ensuring long-term FAIR compliance. Accessing data files via GUPRI would be sensible, but this would be handled by looking up the S3 URI in either Darkroom and/or Catalog (see respective tech reviews).

Open/unclear

Data access APIs

Programmatic access to data is required, via API (S3 API), and/or via eventing systems (most likely native AWS - EventBridge/SQS).

Must-have

Metadata support

The system needs to support annotating data(sets) with minimal metadata (e.g. creation date, version). Detailed metadata collection/management is delegated to the catalog and registration components.

Should-have

Authentication

Compatibility with PingFed and/or Janus authentication. Support for AWS IAM.

Must-have

External data sharing

Need to get clarity on whether the platform will need to be able to share data outside of the Roche network, but currently assuming that this is not required via direct access.

Open/unclear

Rationale for Decision

The following section compares all appropriate options. The gaps are described where only some or few needs are met. A full comparison can be found at the end of the page

Two options were considered to not be viable hence are not included in the

Minerva team has recommended that Minerva S3 buckets are used primarily for application data, and not for scientific data hence it is not included in the comparison or as a viable option

Endor is built on Navify which is intended for customer facing digital products hence is not appropriate for internal usage and so is not included in the comparison or as a viable option

	Data Ecosystem S3 Buckets	RCP S3 Buckets	Arvados
Score	54	51	20
Gold🥇	17	15	6
Silver🥈	1	2	1
Bronze🥉	1	2	0
None	0	0	0

Scoring

Gold 3 points: Most needs are met and close alignment to best practices and guidelines
Silver 2 points: Some needs are met and general alignment to best practices and guidelines
Bronze 1 point: Few needs are met and weak alignment to best practices and guidelines
None 0 points: No needs are met and no alignment to best practices and guidelines

Technical Suitability

Data Ecosystem S3 Buckets: No explicit limitations.
RCP S3 Buckets: Perpetuates a fragmented storage landscape, directly contradicting the core objective of consolidating storage.
Arvados: Enforces rigid internal schemas and governance requirements that dictate a specific filesystem structure, which may not be suitable for the diverse use cases in the PCI platform. There are also concerns regarding its scalability for massively parallel data access.

Engineering Velocity (Time-to-Value) limitations

Data Ecosystem S3 Buckets: Novel integration patterns or support for new regions create a dependency on the Data Ecosystem Storage team’s capacity and roadmap.
RCP S3 Buckets: No explicit limitations.
Arvados: Dependency on the Arvados platform team. Has a dependency on an external open-source community in case certain features are required, which limits flexibility and rapid iteration.

Self-Service Maturity limitations

Data Ecosystem S3 Buckets: No self-service tooling for event configuration, placing the burden on the Data Ecosystem team to enable integrations.
RCP S3 Buckets: Lacks unified governance, tagging, and discoverability; cross-account access patterns and event management must be solved and configured manually per-account.
Arvados: No self service eventing configuration.

Cost limitations

Data Ecosystem S3 Buckets: Moving data from existing storage into these buckets introduces potentially significant migration costs, such as standard data egress/ingress charges, cross-region transfer fees.
RCP S3 Buckets: Poor cost observability and billing attribution out of the box, requiring additional implementation effort.
Arvados: Moving data from existing storage into these buckets introduces potentially significant migration costs, such as standard data egress/ingress charges, cross-region transfer fees.

Operational Complexity limitations

Data Ecosystem S3 Buckets: While providing a managed RCP account and some services, it still requires consumers to establish their own ingestion and sharing components.
RCP S3 Buckets: Places the full burden on consumers by requiring them to manage their own RCP accounts and S3 buckets independently.
Arvados: The added complexity of its events API requires a translation layer to be built and maintained for any S3-native consumers.

AWS Native Integration limitations

Data Ecosystem S3 Buckets: No technical limitations, but non-supported integrations need to be approved and implemented by the Data Ecosystem Storage platform team.
RCP S3 Buckets: None.
Arvados: It is not natively integrated with AWS S3; it uses its own data model and introduces an non-AWS native event abstraction layer on top of raw S3 events.

Compliance limitations

Data Ecosystem S3 Buckets: Currently not aligned with pRED governance processes (like WDIP and disaster recovery SOPs), and pre-baked bucket policy templates that adhere to these rules do not yet exist.
RCP S3 Buckets: Leads to an inconsistent security and compliance posture across accounts. It requires consumers to manage their own audit controls and alignment to Roche policies without support or oversight.
Arvados: No explicit limitations cited in the sources.

Data residency limitations

Data Ecosystem S3 Buckets: Not all RCP-supported regions are currently available via self-service.
RCP S3 Buckets: No explicit limitations cited in the sources.
Arvados: Not all RCP-supported regions are currently available.

Data lifecycle limitations

Data Ecosystem S3 Buckets: Intelligent tiering for deep archive is required but not yet supported.
RCP S3 Buckets: No explicit limitations cited in the sources.
Arvados: No automated tiering is mentioned; to recover previous states of data, collection versioning must be manually enabled.

Data access authorization limitations

Data Ecosystem S3 Buckets: There is currently no published roadmap for supporting Policy Based Access Management (PBAM), and Attribute * Based Access Control (ABAC) is required but not yet supported.
RCP S3 Buckets: Cross-account access patterns must be solved manually per-account.
Arvados: Projects default to private (creator-only) and access must be manually granted to specific users or groups. No integration with native AWS IAM.

Data ownership limitations

Data Ecosystem S3 Buckets: There is currently no way to seamlessly transfer ownership of existing S3 buckets into another AWS account, requiring complex data migrations.
RCP S3 Buckets: No explicit limitations cited in the sources.
Arvados: No explicit limitations cited in the sources.

Data types limitations

Data Ecosystem S3 Buckets: No explicit limitations cited in the sources.
RCP S3 Buckets: No explicit limitations cited in the sources.
Arvados: Its rigid governance limit flexibility for storing the evolving list of diverse file formats and "image-adjacent" data types needed by researchers.

Unique & Persistent Identifiers (GUPRI) limitations

Data Ecosystem S3 Buckets: Requires consumers to implement GUPRIs correctly on their own without support or oversight, leading to inconsistencies and complexities.
RCP S3 Buckets: Requires consumers to implement GUPRIs correctly on their own without support or oversight, leading to inconsistencies and complexities.
Arvados: No limitations cited (it natively supports UUIDs and portable data hashes).

Data access APIs limitations

Data Ecosystem S3 Buckets: No explicit limitations cited in the sources.
RCP S3 Buckets: No centralized event management; each team must configure their integrations independently.
Arvados: It is not a direct pass-through for s3:PutObject events. Events are generated at the collection level rather than per file change, requiring an API translation layer.

Metadata support limitations

Data Ecosystem S3 Buckets: Standard S3 tagging limitations, must be supplemented with data catalog functionality in case these limitations are hit.
RCP S3 Buckets: Offers no unified tagging or metadata schemas out of the box. Standard S3 tagging limitations.
Arvados: Tagging relies on pre-defined keys and values, providing limited flexibility for custom metadata or future-proofing.

Authentication limitations

Data Ecosystem S3 Buckets: No direct integration with PingFed.
RCP S3 Buckets: Not integrated with PingFed or Janus directly.
Arvados: No direct integration with Janus.

External data sharing limitations

Data Ecosystem S3 Buckets: No explicit limitations cited in the sources.
RCP S3 Buckets: No explicit limitations cited in the sources.
Arvados: No explicit limitations cited in the sources.

📚 Resources

Full comparison against criteria

Table 1. Scoring summary
Metric	Data Ecosystem S3 Buckets	RCP S3 Buckets	Arvados
Score	48	42	35
Gold🥇	13	11	7
Silver🥈	4	2	4
Bronze🥉	1	5	6
None	1	1	2

Table 2. Detailed evaluation
Category	Features	Data Ecosystem S3 Buckets	RCP S3 Buckets	Arvados
Technical Suitability	Does it solve the core problem effectively?	🥇	🥉	🥉
Engineering Velocity (Time-to-Value)	Can the feature team deliver this independently, or does it require external dependencies?	🥇	🥇	🥈
Self-Service Maturity	Does this solution allow engineers to pave their own road, for example through Terraform or CDK, or does it require manual ticket filing?	🥇	🥇	🥉
Cost	Is the solution cost-effective, and does it allow costs to be monitored and minimized?	🥇	🥇	🥇
Operational Complexity	Is the maintenance burden on the team minimized?	🥈	🥉	🥇
AWS Native Integration	How well does it leverage existing managed services, thereby reducing operational overhead?	🥉	🥉	🥇
Compliance	The solution must allow data to be stored in a manner aligned with at least the following regulatory and security requirements: GDPR and equivalent local regulations Encryption of data at rest COREMAP requirements for C3 and C4 data Support for the pRED disaster recovery SOP and gRED equivalents DoJ DSP directive German Workers Council requirements RAIA	🥇	🥈	🥇
Data Residency	The system must allow data storage in the United States and Europe.	🥇	🥇	🥈
Data Lifecycle	The system must seamlessly transition data between hot and colder storage tiers to reduce costs where appropriate.	🥇	🥇	🥉
Archiving	The system must provide capabilities for archiving data in compliance with COREMAP requirements.	🥇	🥇	🥇
Data Volume and Velocity	File sizes can vary from kilobytes to terabytes. Larger ingestions currently include approximately 100,000 images representing 300 GB. Ingestion is expected to be limited primarily by the network connection between Roche sites and the cloud storage provider. Analysis may involve thousands of parallel read and write tasks. The solution must support a future increase of 10 to 100 times the current data volume and velocity.	🥇	🥇	🥈
Data Access Authorization	The storage solution must support authorization for access to stored data. Authorization must be determined according to the data-sharing rules defined by RDE WS7 PBAM and any additional applicable criteria.	🥇	🥇	🥈
Data Ownership	The storage solution must attribute data ownership to an individual who is accountable for decisions concerning the data. Ownership must be transferable, for example when an employee changes roles. See the Roche Group Records Management Directive.	🥈	🥉	🥇
Data Types	The PCI platform will store image formats such as TIFF, JPG, NDPI, and Zarr; raw data such as Bruker NMR files; and image-adjacent formats such as segmentation masks and cell detections. The supported formats will evolve as researchers' needs change.	🥇	🥇	🥉
Unique and Persistent Identifiers (GUPRI)	A GUPRI can safeguard data links over time. Catalogued data should be associated with robust, globally unique, and persistent digital identifiers that do not break when data is logically reorganized. Access through GUPRI would be handled by looking up the corresponding S3 URI in Darkroom or the Catalog.	🥈	🥈	🥇
Data Access APIs	Programmatic access to data is required through the S3 API or eventing systems, most likely AWS EventBridge or Amazon SQS.	🥇	🥇	None
Metadata Support	The system must support annotating datasets with minimal metadata, such as creation date and version. Detailed metadata collection and management are delegated to the catalog and registration components.	🥈	🥉	🥉
Authentication	Compatibility with PingFed or Janus authentication and support for AWS IAM are required.	🥇	🥇	🥉
External Data Sharing	Further clarification is required regarding whether the platform must support sharing data outside the Roche network. Direct external access is currently assumed not to be required.	None	None	None