Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
505fd51
Add storage provider abstraction architecture
sven1103 Aug 28, 2026
796437b
docs: apply ISP to storage provider interfaces
KochTobi-Agent Aug 28, 2026
83b2151
docs: move byte-range support to capability interface
KochTobi-Agent Aug 28, 2026
7c9473f
docs: drop getFileSize from StorageProvider
KochTobi-Agent Aug 28, 2026
64c9b5a
docs: make checksum type provider-selected
KochTobi-Agent Aug 28, 2026
bffd37d
docs: key providers by id, type selects required config
KochTobi-Agent Aug 28, 2026
8653ab2
docs: add openbis-nfs hybrid provider type
KochTobi-Agent Aug 28, 2026
99d7f40
Merge main into feature/storage-provider-architecture
KochTobi-Agent Aug 28, 2026
d5602e5
feat: add storage provider abstraction with openbis adapter (#108)
KochTobi Aug 28, 2026
c8a594e
feat: add provider registry and configuration binding (#110)
KochTobi Aug 31, 2026
d4b57fd
feat: add V2 controller with provider abstraction and feature flag
sven1103 Aug 31, 2026
eb24fb4
feat: add OpenBisNfsStorageProvider for hybrid metadata/NFS streaming
sven1103 Aug 31, 2026
0817a17
feat: add NFS test configuration and setup script
sven1103 Aug 31, 2026
85e6d1a
feat: add logging to identify which provider is being used
sven1103 Aug 31, 2026
e189f1c
feat: implement OpenBisNfsStorageProvider with physical path resoluti…
sven1103 Aug 31, 2026
1ba8d9a
Add debug logging to show physical path resolution details
sven1103 Aug 31, 2026
032849e
Add System.out.println for debugging NFS provider
sven1103 Aug 31, 2026
4f39f09
Fix physical path resolution to use sharded structure from openBIS
sven1103 Aug 31, 2026
591e954
Refactor OpenBisNfsStorageProvider to use MeasurementDataProvider int…
sven1103 Aug 31, 2026
70417fd
Refactor OpenBisNfsStorageProvider to use FilePathProvider correctly
sven1103 Aug 31, 2026
66d5f22
feat: support provider-specific openBIS configuration
sven1103 Aug 31, 2026
c98e536
fix: support nested properties in provider configuration
sven1103 Aug 31, 2026
59d5d61
refactor: use typed configuration for provider properties
sven1103 Aug 31, 2026
26c510a
refactor: rename providers map to instances to avoid duplication
sven1103 Aug 31, 2026
9c0ebb0
fix: use actual file sizes from filesystem in NFS provider
sven1103 Aug 31, 2026
3de9c74
feat: add ZIP download endpoint to V2 controller
sven1103 Aug 31, 2026
bcde87f
refactor: rename ignored-prefix to wrapper-directory and fix ZIP file…
sven1103 Aug 31, 2026
9bf2bc3
feat: handle UUID4 task-id in DSS directory structure
sven1103 Aug 31, 2026
6c6df39
docs: external properties file for transparent service config
sven1103-agent Sep 1, 2026
51f2143
docs: document production build in README
sven1103-agent Sep 1, 2026
c31f3a6
refactor: centralize version in a single revision property
sven1103-agent Sep 1, 2026
8dd8bbf
docs: fix external config loading for production
sven1103-agent Sep 1, 2026
3c07da8
fix: throttle download near-full queue warnings to stop log flood
sven1103-agent Sep 1, 2026
83e8103
Remove legacy V1 controller, promote V2 to MeasurementFileController
sven1103-agent Sep 1, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/create-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,11 @@ jobs:
${{ runner.os }}-maven-

- name: Set version in Maven project
run: mvn versions:set -DnewVersion="${{ github.event.inputs.versionTag }}" -DprocessAllModules -DgenerateBackupPoms=false
# Bump the centralized `revision` property in the root POM. With revision-based
# versioning (all modules inherit from ${revision}), the version lives in exactly
# one place. versions:set would overwrite the ${revision} expression, so use
# set-property to update the property value instead.
run: mvn org.codehaus.mojo:versions-maven-plugin:2.18.0:set-property -Dproperty=revision -DnewVersion="${{ github.event.inputs.versionTag }}" -DgenerateBackupPoms=false

- name: Build with Maven
run: mvn -B package --file pom.xml
Expand Down
134 changes: 132 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,132 @@
For the API documentation run the server and visit
http://localhost:8080/swagger-ui/index.html
# Data Download Server

Spring Boot service for downloading (bio)measurement data via the QBiC download-API.

## Configuration

The service is configured via an external `application.properties` file. A commented,
fully-documented template is available at
[`rest-api/src/dist/application.properties`](rest-api/src/dist/application.properties).

### Loading the external file (important)

Spring Boot looks for `application.properties` relative to the **process working directory**
(the directory you are *in* when you launch `java`), **not** relative to the JAR file's own
location. If you just drop the file "next to the JAR" it is very easy for Spring to not find
it — that happens whenever the JAR is launched from a different directory (or under systemd /
a service manager that does not set the working directory).

To make loading deterministic, always pass the file by absolute path on the command line:

```
java -jar /opt/app/rest-server-1.3.0.jar \
--spring.config.additional-location=/opt/app/application.properties
```

Use `--spring.config.additional-location` (not `--spring.config.location`) so the bundled
defaults inside the JAR are still used for any key the external file does not set. The
`file:` prefix is optional when the path is absolute.

Alternatively, when running under systemd, set `WorkingDirectory` so the file is found
relative to it:

```ini
[Service]
WorkingDirectory=/opt/app
ExecStart=/usr/bin/java -jar /opt/app/rest-server-1.3.0.jar
```

### Env vars still work (optional)

Spring Boot's property precedence still applies, so specific values can be overridden without
editing the file, e.g. `SERVER_PORT=9000 java -jar rest-server.jar`. The env var wins over
the external file. The bundled `application.properties` inside the JAR also still supports the
original environment-variable placeholders for backward compatibility.

## Building for production

### Prerequisites

- JDK 21
- Maven 3.x

### Build everything

The project is a multi-module Maven build:

```
data-download-server (POM parent)
├── zip (library)
├── measurement-provider (library)
├── storage-provider (library)
├── openbis-connector (library)
└── rest-api (Spring Boot application "rest-server")
```

The four library modules are **not** standalone services - they are dependencies of the
Spring Boot application in `rest-api`. Building a package compiles every module and bundles
all four libraries into a single executable JAR.

From the project root:

```bash
# Build everything, including tests
mvn package

# Build everything, skip tests (faster, for a release artifact)
mvn package -DskipTests
```

This produces one deployable artifact:

```
rest-api/target/rest-server-<version>.jar
```

### Build only the application (faster)

The `-am` flag ("also make") builds `rest-api` together with all upstream modules it
depends on - so this produces the same deployable JAR, but skips nothing it needs:

```bash
mvn -pl rest-api -am package
```

### Clean rebuild

```bash
mvn -pl rest-api -am clean package
```

### Deploy to Nexus (release)

```bash
mvn deploy
```

Publishes all modules to the QBiC Nexus repository configured in `distributionManagement`.

## Versioning

The whole project uses **one version, defined in one place**: the `<revision>` property in the
root [`pom.xml`](pom.xml). All modules inherit this version and their build artifacts are named
accordingly (e.g. `rest-server-1.3.0.jar`).

To change the project version, edit the single property:

```xml
<properties>
<revision>1.3.0</revision>
</properties>
```

There is no per-module `<version>` anywhere - child POMs reference the parent via
`${revision}` and inter-module dependencies via `${project.version}`, so a bump is
applied consistently to every module in one edit. The release workflow bumps this property
automatically (`.github/workflows/create-release.yml`).

## API documentation

Run the server and visit
[http://localhost:8090/swagger-ui.html](http://localhost:8090/swagger-ui.html)
(port depends on your configuration; default `8090`).
3 changes: 1 addition & 2 deletions measurement-provider/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,11 @@
<parent>
<groupId>life.qbic</groupId>
<artifactId>data-download-server</artifactId>
<version>1.0.10</version>
<version>${revision}</version>
</parent>

<groupId>life.qbic.data-download</groupId>
<artifactId>measurement-provider</artifactId>
<version>1.0.10</version>
<packaging>jar</packaging>
<dependencies>
<dependency>
Expand Down
10 changes: 7 additions & 3 deletions openbis-connector/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,12 @@
<parent>
<groupId>life.qbic</groupId>
<artifactId>data-download-server</artifactId>
<version>1.0.10</version>
<version>${revision}</version>
</parent>

<groupId>life.qbic.data-download</groupId>
<artifactId>openbis-connector</artifactId>
<packaging>jar</packaging>
<version>1.0.10</version>

<dependencyManagement>
<dependencies>
Expand Down Expand Up @@ -66,7 +65,12 @@
<dependency>
<groupId>life.qbic.data-download</groupId>
<artifactId>measurement-provider</artifactId>
<version>1.0.10</version>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>life.qbic.data-download</groupId>
<artifactId>storage-provider</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,11 @@ public class OpenBisConnector implements MeasurementFinder, MeasurementDataProvi

private static final String UUID_REGEX = "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$";

/**
* Constructor for creating provider-specific instances with custom configuration.
* Used by ProviderRegistryConfig to create openBIS connectors with provider-specific settings.
* Also used as the Spring-injected constructor for the global openBIS connector bean.
*/
public OpenBisConnector(
@Qualifier("openbisSessionFactory") SessionFactory sessionFactory,
@Value("${openbis.server.application.url}") String applicationServerUrl,
Expand Down Expand Up @@ -147,13 +152,20 @@ private List<DataSetFile> searchFilesForMeasurement(OpenBisSession session,
.toList();
}

public List<DataSet> loadDataSetsForMeasurement(MeasurementId measurementId) {
try (var session = sessionFactory.getSession()) {
return loadDataSetsForMeasurement(session, measurementId);
}
}

private List<DataSet> loadDataSetsForMeasurement(OpenBisSession session,
MeasurementId measurementId) {
DataSetSearchCriteria dataSetSearchCriteria = new DataSetSearchCriteria();
dataSetSearchCriteria.withSample().withCode().thatEquals(measurementId.id());

DataSetFetchOptions dataSetFetchOptions = new DataSetFetchOptions();
dataSetFetchOptions.withChildrenUsing(dataSetFetchOptions);
dataSetFetchOptions.withPhysicalData();

return applicationServer.searchDataSets(session.getToken(),
dataSetSearchCriteria,
Expand Down
Loading
Loading