vfps

A very fast and resource-efficient pseudonym service.

Supports horizontal service replication for highly-available deployments.

Run it

Warning Using the provided docker-compose.yaml is not a production-ready deployment but merely used to get started and testing quickly. It sets very restrictive resource limits uses the default password for an included, unoptimized PostgreSQL deployment.

docker compose -f compose.yaml --profile=test up

Visit http://localhost:8080/ to view the OpenAPI specification of the Vfps API:

You can use the JSON-transcoded REST API described via OpenAPI or interact with the service using gRPC. For example, using grpcurl to create a new namespace:

grpcurl \
  -plaintext \
  -import-path src/Vfps/ \
  -proto src/Vfps/Protos/vfps/api/v1/namespaces.proto \
  -d '{"name": "test", "pseudonymGenerationMethod": "PSEUDONYM_GENERATION_METHOD_SECURE_RANDOM_BASE64URL_ENCODED", "pseudonymLength": 32}' \
  127.0.0.1:8081 \
  vfps.api.v1.NamespaceService/Create

And to create a new pseudonym inside this namespace:

grpcurl \
  -plaintext \
  -import-path src/Vfps/ \
  -proto src/Vfps/Protos/vfps/api/v1/pseudonyms.proto \
  -d '{"namespace": "test", "originalValue": "to be pseudonymized"}' \
  127.0.0.1:8081 \
  vfps.api.v1.PseudonymService/Create

Production-grade deployment

See https://github.com/miracum/charts/tree/master/charts/vfps for a production-grade deployment on Kubernetes via Helm.

Configuration

Available configuration options which can be set as environment variables:

Variable	Type	Default	Description
ConnectionStrings__PostgreSQL	string	""	Connection string to the PostgreSQL database. See https://www.npgsql.org/doc/connection-string-parameters.html for options. Also used, when Authorization__IsEnabled=true, to persist the ASP.NET Core Data Protection key ring so auth cookies/antiforgery tokens minted by one replica remain valid on another - see the efbundle-dataprotection migrations bundle.
ForceRunDatabaseMigrations	bool	false	Run database migrations as part of the startup. Only recommended when a single replica of the application is used.
Tracing__IsEnabled	bool	false	Enable distributed tracing support.
Tracing__ServiceName	string	"vfps"	Tracing service name.
Tracing__RootSampler	string	"AlwaysOnSampler"	Tracing parent root sampler. One of AlwaysOnSampler, AlwaysOffSampler, TraceIdRatioBasedSampler
Tracing__SamplingProbability	double	0.1	Sampling probability to use if Tracing__RootSampler is set to TraceIdRatioBasedSampler.
Tracing__Otlp__Endpoint	string	""	The OTLP gRPC Endpoint URL traces are exported to (e.g. a Jaeger v2 or other OTLP-compatible collector).
Pseudonymization__Caching__Namespaces__IsEnabled	bool	false	Set to true to enable namespace caching.
Pseudonymization__Caching__Pseudonyms__IsEnabled	bool	false	Set to true to enable pseudonym caching.
Pseudonymization__Caching__SizeLimit	int	65534	Maximum number of entries in the cache. The cache is shared between the pseudonyms and namespaces.
Pseudonymization__Caching__AbsoluteExpiration	D.HH:mm:nn	0.01:00:00	Time after which a cache entry expires.
Authorization__IsEnabled	bool	false	Enable OIDC authentication and namespace-scoped authorization for the API and admin UI. Off by default: with this disabled, the API and admin UI are reachable without authentication and every namespace is fully accessible. Note for deployments running the bundled migrations as a separate init job: run both efbundle and efbundle-dataprotection when this is enabled.
Authorization__Authority	string	""	The OIDC issuer/authority URL (e.g. a Keycloak realm URL).
Authorization__Audience	string	""	Audience validated for bearer-token (gRPC/REST API) callers.
Authorization__ClientId	string	""	Confidential client id used by the admin UI's Authorization Code flow.
Authorization__ClientSecret	string	""	Confidential client secret used by the admin UI's Authorization Code flow.
Authorization__RoleClaimType	string	"roles"	Which claim on the validated token carries the caller's roles/groups.
Authorization__AdminRoles__0, __1, ...	string	-	Roles granting full access: all namespaces, plus namespace create/delete.
Authorization__NamespaceRules__0__Namespace	string	-	Namespace name this rule applies to, or "*" for all namespaces.
Authorization__NamespaceRules__0__ReadRoles__0, ...	string	-	Roles granting read access (namespace browsing, pseudonym list) to the namespace above.
Authorization__NamespaceRules__0__WriteRoles__0, ...	string	-	Roles granting write access to the namespace above.
Authorization__NamespaceRules__0__ReverseLookupRoles__0, ...	string	-	Roles granting reverse-lookup access (revealing a pseudonym's original value) to the namespace above - a separate, more tightly-scoped grant than read access.

NamespaceRules as indexed env vars gets unwieldy for more than a couple of namespaces; mounting a JSON file (e.g. via appsettings.Production.json or a config provider pointed at a mounted file) for this section is a reasonable alternative for larger rule sets.

Variable	Type	Default	Description
S3__IsEnabled	bool	false	Enable CSV pseudonymization jobs (admin UI upload/download + the Hangfire job runner). Off by default - also requires ConnectionStrings__PostgreSQL (Hangfire reuses the same database for its job storage).
S3__ServiceUrl	string	""	S3-compatible endpoint URL, e.g. a local MinIO instance or a real AWS/S3-compatible endpoint.
S3__AccessKey	string	""	Access key for the bucket above.
S3__SecretKey	string	""	Secret key for the bucket above.
S3__Bucket	string	""	Bucket CSV job input/output files are stored in.
S3__Region	string	"us-east-1"	Region passed to the S3 client.
S3__ForcePathStyle	bool	true	Path-style addressing (https://host/bucket/key) rather than virtual-hosted-style - required for MinIO and most non-AWS S3-compatible stores.
S3__PresignedUrlExpiry	TimeSpan	"0.00:15:00"	How long presigned upload/download URLs remain valid.
S3__ObjectRetentionDays	int	30	Days before an S3 bucket lifecycle rule expires (deletes) a CSV job's input/output objects. Applied on startup, scoped to the csv-jobs/ prefix. Job records themselves are never deleted, so without this the original, unpseudonymized input would otherwise live in the bucket forever. Set to 0 to leave the bucket's lifecycle configuration untouched. Overwrites the bucket's entire existing lifecycle configuration - use a bucket dedicated to vfps.

CSV job input/output bytes never pass through the vfps process itself: the admin UI uploads directly to a presigned S3 PUT URL and downloads directly from a presigned S3 GET URL, and the Hangfire background job (running in-process, no separate worker deployment) streams the file S3-to-S3. See compose.yaml's s3 profile for a local MinIO setup usable for manual testing.

Each job runs in one of two directions: Pseudonymize (replace original values with their pseudonym - requires write access to every namespace used) or De-pseudonymize (replace pseudonym values with their original value - requires reverse-lookup access, since this reveals data). A pseudonym with no match in its namespace during de-pseudonymization is left unchanged in the output rather than failing the job.

Observability

The service exports metrics in Prometheus format on :8082/metrics. Health-, readiness-, and liveness-probes are exposed at :8080/healthz, :8080/readyz, and :8080/livez respectively.

FHIR operations

The service also exposes a FHIR operations endpoint. Sending a FHIR Parameters resource to /v1/fhir/$create-pseudonym of the following schema:

{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "namespace",
      "valueString": "test"
    },
    {
      "name": "originalValue",
      "valueString": "hello world"
    }
  ]
}

will create a pseudonym in the test namespace. The expected response looks as follows:

{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "namespace",
      "valueString": "test"
    },
    {
      "name": "originalValue",
      "valueString": "hello world"
    },
    {
      "name": "pseudonymValue",
      "valueString": "8KWwnm3TXR5R9iUDVVKD-jUezE4DEyeydOeq4v_a_b5ejSLmqOlT8g"
    }
  ]
}

Development

Prerequisites

• .NET 7.0: https://dotnet.microsoft.com/en-us/download/dotnet
• Docker CLI 20.10.17: https://www.docker.com/
• Docker Compose: https://docs.docker.com/compose/install/

Build & run

Start an empty PostgreSQL database, Keycloak, and MinIO for development (optionally add -d to run in the background):

docker compose -f compose.yaml --profile=keycloak --profile=s3 up

To additionally start an instance of Jaeger Tracing, you can specify the jaeger profile:

docker compose -f compose.yaml --profile=jaeger up

Then set Tracing__IsEnabled=true and Tracing__Otlp__Endpoint=http://localhost:4317 (already the default in appsettings.Development.json) and view traces at http://localhost:16686.

Restore dependencies and run in Debug mode:

dotnet restore
dotnet run -c Debug --project=src/Vfps

Open https://localhost:8080/ to see the OpenAPI UI for the JSON-transcoded gRPC services. You can also use grpcurl to interact with the API:

Note In development mode gRPC reflection is enabled and used by grpcurl by default.

grpcurl -plaintext \
    -d '{"name": "test", "pseudonymGenerationMethod": "PSEUDONYM_GENERATION_METHOD_SECURE_RANDOM_BASE64URL_ENCODED", "pseudonymLength": 32}' \
    127.0.0.1:8081 \
    vfps.api.v1.NamespaceService/Create

grpcurl -plaintext \
    -d '{"namespace": "test", "originalValue": "a test value"}' \
    127.0.0.1:8081 \
    vfps.api.v1.PseudonymService/Create

Run unit tests

dotnet test src/Vfps.Tests \
  --configuration=Release \
  --results-directory=./coverage \
  -- --coverage \
  --coverage-output-format cobertura \
  --coverage-output coverage.cobertura.xml \
  --coverage-settings src/Vfps.Tests/codecoverage.config

Generate Code coverage report

If not installed, install the report generation too:

dotnet tool install -g dotnet-reportgenerator-globaltool

reportgenerator -reports:"./coverage/coverage.cobertura.xml" -targetdir:"coveragereport" -reporttypes:Html

Build container image

docker build -t ghcr.io/miracum/vfps:latest .

Benchmarks

Micro benchmarks

The pseudonym generation methods are continuously benchmarked. Results are viewable at https://miracum.github.io/vfps/dev/bench/.

E2E load testing

Create a pseudonym namespace used for benchmarking:

grpcurl \
  -plaintext \
  -import-path src/Vfps/ \
  -proto src/Vfps/Protos/vfps/api/v1/namespaces.proto \
  -d '{"name": "benchmark", "pseudonymGenerationMethod": "PSEUDONYM_GENERATION_METHOD_SECURE_RANDOM_BASE64URL_ENCODED", "pseudonymLength": 32}' \
  127.0.0.1:8081 \
  vfps.api.v1.NamespaceService/Create

Generate 100.000 pseudonyms in the namespace from random original values:

ghz -n 100000 \
    --insecure \
    --import-paths src/Vfps/ \
    --proto src/Vfps/Protos/vfps/api/v1/pseudonyms.proto \
    --call vfps.api.v1.PseudonymService/Create \
    -d '{"originalValue": "{{randomString 32}}", "namespace": "benchmark"}' \
    127.0.0.1:8081

Sample output running on

OS=Windows 11 (10.0.22000.978/21H2)
12th Gen Intel Core i9-12900K, 1 CPU, 24 logical and 16 physical cores
32GiB of DDR4 4800MHz RAM
Samsung SSD 980 Pro 1TiB
PostgreSQL running in WSL2 VM on the same machine.
.NET SDK=7.0.100-rc.1.22431.12

Summary:
  Count:        100000
  Total:        16.68 s
  Slowest:      187.81 ms
  Fastest:      2.52 ms
  Average:      8.00 ms
  Requests/sec: 5993.51

Response time histogram:
  2.522   [1]     |
  21.051  [99748] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  39.580  [201]   |
  58.109  [0]     |
  76.639  [0]     |
  95.168  [0]     |
  113.697 [0]     |
  132.226 [0]     |
  150.755 [0]     |
  169.285 [0]     |
  187.814 [50]    |

Latency distribution:
  10 % in 6.26 ms
  25 % in 6.91 ms
  50 % in 7.72 ms
  75 % in 8.93 ms
  90 % in 9.57 ms
  95 % in 10.01 ms
  99 % in 11.86 ms

Status code distribution:
  [OK]   100000 responses

Sub-10ms P99-latency

By default, each pseudonym creation requests executes two database queries: one to fetch the namespace configuration and a second one to persist the pseudonym if it doesn't already exist. There is an opt-in way to avoid the first query by caching the namespaces in a non-distributed in-memory cache. It can be enabled and configured using the following environment variables:

Variable	Type	Default	Description
Pseudonymization__Caching__Namespaces__IsEnabled	bool	false	Set to true to enable namespace caching.
Pseudonymization__Caching__SizeLimit	int	32	Maximum number of entries in the cache.
Pseudonymization__Caching__AbsoluteExpiration	D.HH:mm:nn	0.01:00:00	Time after which a cache entry expires.

Warning Deleting a namespace does not automatically remove it from the in-memory cache. Pseudonym creation requests against such a stale cached namespace will fail until either the entry expired or the service is restarted.

Using the same setup as above but with namespace caching enabled, we can lower the per-request latencies and increase throughput:

Summary:
  Count:        100000
  Total:        11.70 s
  Slowest:      27.23 ms
  Fastest:      1.55 ms
  Average:      5.47 ms
  Requests/sec: 8549.22

Response time histogram:
  1.546  [1]     |
  4.114  [17418] |∎∎∎∎∎∎∎∎∎∎
  6.682  [72827] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  9.251  [9382]  |∎∎∎∎∎
  11.819 [122]   |
  14.387 [0]     |
  16.956 [0]     |
  19.524 [0]     |
  22.092 [0]     |
  24.661 [49]    |
  27.229 [201]   |

Latency distribution:
  10 % in 4.00 ms
  25 % in 4.34 ms
  50 % in 5.81 ms
  75 % in 6.00 ms
  90 % in 6.66 ms
  95 % in 7.00 ms
  99 % in 8.00 ms

Status code distribution:
  [OK]   100000 responses

Resource efficiency

The sample deployment described in docker-compose.yaml sets strict resource limits for both the CPU (1 CPU) and memory (max 128MiB). Even under these constraints > 1k RPS are possible, although with significantly increased P99 latencies:

Summary:
  Count:        100000
  Total:        73.99 s
  Slowest:      268.06 ms
  Fastest:      5.26 ms
  Average:      36.69 ms
  Requests/sec: 1351.51

Response time histogram:
  5.257   [1]     |
  31.537  [57298] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  57.817  [21327] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  84.097  [17685] |∎∎∎∎∎∎∎∎∎∎∎∎
  110.377 [3395]  |∎∎
  136.656 [243]   |
  162.936 [0]     |
  189.216 [1]     |
  215.496 [0]     |
  241.776 [0]     |
  268.055 [50]    |

Latency distribution:
  10 % in 14.62 ms
  25 % in 18.47 ms
  50 % in 29.46 ms
  75 % in 47.53 ms
  90 % in 71.96 ms
  95 % in 79.95 ms
  99 % in 97.22 ms

Status code distribution:
  [OK]   100000 responses

Image signature and provenance verification

Prerequisites:

• cosign
• slsa-verifier
• crane

All released container images are signed using cosign and SLSA Level 3 provenance is available for verification.

IMAGE=ghcr.io/miracum/vfps:v1.4.3
DIGEST=$(crane digest "${IMAGE}")
IMAGE_DIGEST_PINNED="ghcr.io/miracum/vfps@${DIGEST}"
IMAGE_TAG="${IMAGE#*:}"

cosign verify \
   --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
   --certificate-identity-regexp="https://github.com/miracum/.github/.github/workflows/standard-build.yaml@.*" \
   --certificate-github-workflow-name="ci" \
   --certificate-github-workflow-repository="miracum/vfps" \
   --certificate-github-workflow-trigger="release" \
   --certificate-github-workflow-ref="refs/tags/${IMAGE_TAG}" \
   "${IMAGE_DIGEST_PINNED}"

slsa-verifier verify-image \
    --source-uri github.com/miracum/vfps \
    --source-tag ${IMAGE_TAG} \
    "${IMAGE_DIGEST_PINNED}"

See also https://github.com/slsa-framework/slsa-github-generator/tree/main/internal/builders/container#verification for details on verifying the image integrity using automated policy controllers.