Test your LLM apps like the rest of your Java code

How it works

The test structure you already know

1given()
Describe the case
A question, the answer you expect, plus the context or tool calls it should use.
2when()
Call your app
Pass any function: a LangChain4j AI service, a REST client, your own pipeline.
3then()
Assert on quality
Set score thresholds per metric. A regression fails the build like any other test.

Metrics

Measure what matters in your LLM app

LLM judge

Answer Correctness

Compares the claims in an answer against your ground truth.

Faithfulness

Catches hallucinations: is every claim backed by the given context?

Answer Relevance

Does the answer actually address the question that was asked?

Semantic Similarity

Embedding distance between the answer and a reference answer.

Tool Call Accuracy

Verifies your agent called the right tools with the right arguments.

BLEU & ROUGE

Classic n-gram and LCS overlap scores, fast and free to run.

Modules

Pick only what you need

rage4j

Core

Evaluators, samples and aggregation. Use it anywhere, not only in tests.

rage4j-assert

Assert

The fluent given / when / then API for LLM assertions in your test suite.

rage4j-persist

Persist

Write every evaluation to JSON Lines to track quality across runs.

rage4j-persist-junit5

Persist JUnit 5

One annotation manages the store lifecycle for a whole test class.

Installation

Add one test dependency

Rage4J is on Maven Central. It builds on LangChain4j, so bring the model provider you already use, such as OpenAI or Ollama.

Maven
Gradle

<dependency>
    <groupId>dev.rage4j</groupId>
    <artifactId>rage4j-assert</artifactId>
    <version>2.0.2</version>
    <scope>test</scope>
</dependency>

testImplementation("dev.rage4j:rage4j-assert:2.0.2")

Stop eyeballing LLM output.

Write your first evaluation test in five minutes.

Read the docs →See examples