LLM-as-a-Judge: The Importance of Harshness

Editor’s Note

At its heart, this piece tackles a problem that’s becoming more relevant as AI tools spread into everyday work: how do you check the quality of something an AI produced, when there’s no answer key to grade against? Dr. Michael Overton’s answer (using one AI model to evaluate another’s output) is a clever workaround, but the real discovery here is more subtle and more useful. It turns out that how you ask the AI judge to evaluate matters enormously. When prompted to simply assess quality, the AI grader tended to hand out generous, feel-good scores across the board, the way a polite colleague might soften their feedback in a performance review. But when nudged to act as a strict, critical evaluator, the same AI began drawing sharper distinctions between good and mediocre work, surfacing flaws it had previously glossed over. The takeaway isn’t just academic: anyone using AI to review writing, summarize documents, or check other AI-generated content should know that the instructions you give the evaluator can quietly shape the verdict. A small tweak in tone, “be a tough critic” versus “give your assessment,” can be the difference between useful feedback and empty praise.

Evaluating LLMs with…LLMs:

I have a growing fascination with evaluating the outputs of Large Language Models (LLMs). The core question is simple, yet daunting: How do we evaluate complex natural language outputs at scale? The most trusted evaluator is a human expert, but experts are difficult to find, expensive, and human-in-the-loop evaluation practices don’t scale. There is an added challenge when the “correctness” of a requested LLM response is subjective. If there is ground truth, then we can evaluate the accuracy of an LLM (i.e., how often it is correct). However, when there is no true “correct” response, even human evaluators can inject substantial bias into their assessments.  

One promising solution is an approach called LLM-as-a-Judge​ (Zheng, 2023)​ where we use LLMs to judge LLMs. It is an elegant idea and scalable solution to the evaluation problem as LLM judges can be prompted for a variety of different evaluation tasks. Further, this approach solves evaluation concerns when the “correctness” of LLM outputs is either unknown (in the case of data without labels) or unknowable (in the case of outputs like summaries). While this method introduces a number of known biases, those are beyond the scope of this blog post. 

Background

I am very interested in using multiple LLM judges to provide wholistic evaluations of LLM outputs so, I decided to run a small test evaluating LLM sentiment analysis results from a side-project. Sentiment analysis is somewhat subjective (coders look at a text and decide if it expresses a positive, neutral, or negative sentiment). I took a random sample of 400 observations and selected the outputs of 4 models to compare the results. I used GPT-OSS with 120 billion parameters as the judge model. 

Models being Evaluated

1. Command A (best performing model on the sentiment task)
2. GPT-oss (High performing thinking model)
3. Phi 4 (small, yet relatively high performing model)
4. Gemma 3 (underperforming model for comparative purposes) 

Judge Prompt and Criteria 

I used the G-eval prompt ​(Liu, 2023)​ as the foundation for the judge prompt and I made individual prompts for each of the following 9 criteria: 

• Fluency
• Coherence
• Plausibility
• Entailment
• Factuality
• Faithfulness
• Relevance
• Informativeness
• Restatement

def prompt_coherence_llm_judge(explanation): 
    prompt = f""" 
### Task Overview: You will be given a User Generated Explanation and your objective is to provide a Coherence score. Carefully review all instructions before replying. 
 
### User Generated Explanation: {explanation} 
 
### Evaluation Task:  
Using the User Generated Explanation, answer the following question: 
Coherence: How well do the ideas flow logically from one sentence to the next in this response? Respond by selecting 1 item from the scale that most closely matches your evaluation of the User Explanation and your Rationale for that decision. Provide your Rationale before your scale answer, 
A) No logical flow between sentences - Most sentence pairs show no relation to one another.  
B) Minimal logical flow between sentences - Only a few sentence pairs show minimal connection. 
C) Moderate logical flow between sentences - Many sentence pairs show moderate connection with occasional gaps. 
D) Strong logical flow between sentences - The majority of sentence pairs display clear, purposeful connections. 
E) Excellent logical flow between sentences – Every sentence seamlessly builds on the previous one, forming a cohesive argument. 
 
### Instructions:  
1. Read and thoroughly understand the Task Overview, User Generated Explanation, Evaluation Task and Instructions.  
2. Ensure that your output is in valid JSON format {{Rationale: <"Insert Rationale Here">, Coherence: <No logical flow between sentences | Minimal logical flow between sentences |Moderate logical flow between sentences |Strong logical flow between sentences |Excellent logical flow between sentences>}} 
3. Do not make up reasons or find fault where there is none, but your evaluations should be very critical. 
4. **Read the Explanation sequentially**  
    – Examine each sentence in the order presented, noting the logical connections (e.g., use of transition words, references to previous ideas).  
5. **Determine sentence‑to‑sentence relationships**  
    – For each pair of consecutive sentences, ask: *Does the second sentence logically follow from, elaborate on, or contrast with the first?* Record a brief note.  
6. **Select the Appropriate Rating** 
   - Based on the cumulative assessment of Coherence identified in step 4, choose the Scale item that best reflects the overall Coherence of the Explanation. 
7. **Structure the Reasoning for the Final Score** 
   - a. Write a concise Rationale for your decision that references specific sentence links. Return your response with the following key-value pair {{Rationale: <"Insert Rationale Here">}}. 
   - b. Provide the selected scale rating based ONLY on your documented Rationale. Ensure that your previously stated Rationale aligns with the Scale item description. Return your response with the following key-value pair {{Coherence: <No logical flow between sentences | Minimal logical flow between sentences |Moderate logical flow between sentences |Strong logical flow between sentences |Excellent logical flow between sentences>}}. 
8. Review your responses for accuracy. Ensure that your Rationale and Scale item is accurate and about the User Generated Explanation. Change it if corrections are required.  
9. Your final output MUST be a strict JSON object. Do not include any introductory or concluding text, conversational statements, or markdown text outside of the JSON object. 
 
### Output Format:  
{{Rationale: <"Insert Rationale Here">,  
Coherence: <No logical flow between sentences | Minimal logical flow between sentences |Moderate logical flow between sentences |Strong logical flow between sentences |Excellent logical flow between sentences>}}""" 
    return prompt 

The results from these prompts were less than encouraging. They suggested lots of very ‘nice’ judgements, which you can view in the next section. Great when practicing emotional intelligence, and awful when trying to identify nuanced flaws/differences. 

Harsh Evaluator

As an experiment, I added a few lines to the judge’s prompts doing something I generally stay away from–role prompting. Basically, I added three lines prompting the LLM to be a “harsh but fair evaluator”. Relatively minor, but it made all the difference. The full prompt can be found in the appendix. 

You can see the distributions of the scores between the harsh and not harsh prompts differ in meaningful and statistically significant ways. Harsh prompts have wider distributions of scores meaning that we got an increased breadth of responses from the judge model.

GPT-OSS: The Outlier

I purposefully included GPT-OSS as both the judge model and the model to be judged since judge models tend to show favorable bias toward their responses. There are two odd results I wanted to highlight and one result before I forget it. GPT-OSS was much harsher on itself than other models. Below I show the average responses and I chart the average difference between the Not Harsh response and the Harsh response. GPT- OSS judges itself very differently than other models, and surprisingly it seems to overreact when responding to the harsh prompt.

Does this hold with a larger sample? Will it hold across models or is this true for only GPT- OSS? I’m not sure, but it is something to keep an eye on.

def prompt_coherence_llm_judge2(explanation):
    prompt = f"""
### Task Overview: Act as an extremely harsh, but fair evaluator. You will be given a User Generated Explanation and your objective is to provide a Coherence score. Carefully review all instructions before replying.

### User Generated Explanation: {explanation}

### Evaluation Task: 
Using the User Generated Explanation, answer the following question:
Coherence: How well do the ideas flow logically from one sentence to the next in this response? Respond by selecting 1 item from the scale that most closely matches your evaluation of the User Explanation and your Rationale for that decision. Provide your Rationale before your scale answer,
A) No logical flow between sentences - Most sentence pairs show no relation to one another. 
B) Minimal logical flow between sentences - Only a few sentence pairs show minimal connection.
C) Moderate logical flow between sentences - Many sentence pairs show moderate connection with occasional gaps.
D) Strong logical flow between sentences - The majority of sentence pairs display clear, purposeful connections.
E) Excellent logical flow between sentences – Every sentence seamlessly builds on the previous one, forming a cohesive argument.

### Instructions: 
1. Read and thoroughly understand the Task Overview, User Generated Explanation, Evaluation Task and Instructions. 
2. Ensure that your output is in valid JSON format {{Rationale: <"Insert Rationale Here">, Coherence: <No logical flow between sentences | Minimal logical flow between sentences |Moderate logical flow between sentences |Strong logical flow between sentences |Excellent logical flow between sentences>}}
3. Before proceeding, remember that you are to respond as a harsh critic. Do not make up reasons or find fault where there is none, but your evaluations should be very critical.
4. **Read the Explanation sequentially** 
    – Examine each sentence in the order presented, noting the logical connections (e.g., use of transition words, references to previous ideas). 
5. **Determine sentence‑to‑sentence relationships** 
    – For each pair of consecutive sentences, ask: *Does the second sentence logically follow from, elaborate on, or contrast with the first?* Record a brief note. 
6. **Select the Appropriate Rating**
   - Based on the cumulative assessment of Coherence identified in step 4, choose the Scale item that best reflects the overall Coherence of the Explanation.
7. **Structure the Reasoning for the Final Score**
   - a. Write a concise Rationale for your decision that references specific sentence links. Return your response with the following key-value pair {{Rationale: <"Insert Rationale Here">}}.
   - b. Provide the selected scale rating based ONLY on your documented Rationale. Ensure that your previously stated Rationale aligns with the Scale item description. Return your response with the following key-value pair {{Coherence: <No logical flow between sentences | Minimal logical flow between sentences |Moderate logical flow between sentences |Strong logical flow between sentences |Excellent logical flow between sentences>}}.
8. Review your responses for accuracy. Ensure that your Rationale and Scale item is accurate and about the User Generated Explanation. Change it if corrections are required. 
9. Review your responses to ensure you are a harsh but fair evaluator. Be critical and honest with your evaluation. Change your response if necessary.
10. Your final output MUST be a strict JSON object. Do not include any introductory or concluding text, conversational statements, or markdown text outside of the JSON object.

### Output Format: 
{{Rationale: <"Insert Rationale Here">, 
Coherence: <No logical flow between sentences | Minimal logical flow between sentences |Moderate logical flow between sentences |Strong logical flow between sentences |Excellent logical flow between sentences>}}"""
    return prompt