Gemma2 vs Qwen2 vs Mistral Nemo vs...

Testing logical fallacy detection

Page content

Recently we have seen several new LLMs were released. Exciting times. Let’s test and see how they perform when detecting logical fallacies.

Presenting mistral nemo

What and how we do

In previous test we already tested how LLM are detecting logical fallacies - llama3 vs phi3 vs others. They do it quite well, we loved the result of llama3-8b-Q8 and phi3-medium-4k-instruct-q6_K.

In this test we do the same using Gemma2, Qwen2 and Mistral Nemo.

We are testing the best models possible, but they need to meet our restrictions - they have to fit int our consumer-grade GPU, which has 16 GB of VRAM. With this criteria in mind we picked the following quantizations:

  • gemma2:9b-instruct-q8_0
  • gemma2:27b-instruct-q3_K_L
  • mistral-nemo:12b-instruct-2407-q8_0
  • qwen2:7b-instruct-q8_0

and this qwen2 version:

  • qwen2:72b-instruct-q2_K

It has a size 29GB, but let’s test it anyway.

Please notice here we are not falling back using ‘default’ quantizaton q4. if q8 fits into our GPU’s 16GB VRAM - we are going for it, trying best possible in our circustamces.

DT;DR: The result

Previous models we tested in Logical Fallacy Detection with LLMs. Please see in that post what we test and now.

Here is updated table where our new models are in bold.

ModelVersion GPU RAM A2CS Straw man Ad homi-nem Red Herring Appeal to igno-rance Appeal to autho-rity False dicho-tomy Hasty Gene-rali-zation Begging the Ques-tion Slippery Slope Post hoc ergo propter hoc Tu quo-que Other
l3:8b-i-q4_0 5.8GB V V V V V V
l3:8b-i-q8_0 9.1GB V V V V V V
l3:8b-i-fp16 9.1GB V V V V V
p3:3.8b 4.3GB V V V V V V
p3:3.8b-mini-4k-i-q8_0 5.8GB V V V V V V V V V
p3:3.8b-mini-i-4k-fp16 8.9GB V V V V V V
p3:14b 9.3GB V V V V
p3:14b-m-4k-i-q6_K 12.5GB V V V V V V
p3:14b-m-4k-i-q8_0 15.3GB V V V V V
mi:7b-i-v0.3-q4_0 4.3GB V V V V V V V V V
mi:7b-i-v0.3-q8_0 8.6GB V V V V V V
mi:7b-i-v0.3-fp16 14.7GB V V V
ge:7b-i-v1.1-q4_0 7.2GB V V V V
ge:7b-i-v1.1-q6_K 9.1GB V V V V V
ge:7b-i-v1.1-fp16 15.5GB V V V V V
qw:14b-chat-v1.5-q8_0 15.2GB V V V V V
qw:7b-chat-v1.5-fp16 15.5GB V V V V
qw:32b-chat-v1.5-q3_K_S 15.1GB V V V V
qw:32b-chat-v1.5-q3_K_M 15.4GB V V V V V V V V
ge2:9b-i-q8_0 9.0GB V V V V
ge2:27b-i-q3_K_L 14.0GB V V V
mi-nemo:12b-i-.q8_0 13.0GB V V V V
qw2:7b-i-q8_0 8.1GB V V V V V V
qw2:72b-i-q2_K 29.0GB V V V

Shortcuts in Logical Fallacy names: A 2 CS=Appeal to common sense

Shortcuts in model names:

  • l3=llama3
  • p3=phi3
  • i=instruct
  • m=medium
  • mi=mistral
  • ge=gemma
  • qw=qwen
  • ge2=gemma2
  • qw2=qwen2

Oversimplification=False dichotomy

Conclusion

We expect LLMs to find in sample text the Strawman fallacy, Ad Hominem, Appeal to authority and Oversimplification.

Most new models can see in our sample text Appeal to Authority, and some see Ad Hominem, some Oversimplification. Good results showed qwen2:7b-instruct-q8_0 and gemma2:9b-instruct-q8_0.

But still, winers remain the same.

The “Editor’s choice” model is phi3:14b-medium-4k-instruct-q6_K. It easily fits into 16GB RAM, and quite smart. The second place is for phi3:3.8b-mini-4k-instruct-q8_0. The group of llama3 models is good also.

Mistral Nemo

Let’s review Mistral Nemo results first, even they are not the best, because they have nice-looking robot on their image.

Mistral nemo 12b

Model: mistral-nemo:12b-instruct-2407-q8_0

Response:

The phrase you provided contains two main logical fallacies:

  1. Appeal to Authority (Argumentum ad Verecundiam): The speaker appeals to the authority of a former home affairs minister without providing any specific arguments or evidence from this person. They simply assume that because this person held a certain position, their opinion should be accepted as valid.

    • Fallacy: “as a former home affairs minister should know.”

  2. Red Herring (Irrelevant Conclusion): The speaker starts by presenting a seemingly reasonable argument (“too many people, not enough houses”) but then dismisses it without providing any counterarguments or evidence. Instead, they appeal to the authority of the former minister.

    • Fallacy: “Look, on first blush, it all sounds perfectly reasonable: too many people, not enough houses. But it is never that simple…”

While the phrase does not explicitly use emotional language, the appeal to authority and the red herring fallacies can still be considered forms of appeal to emotion in a broader sense, as they try to persuade the listener based on factors other than rational argumentation.

Based on the phrase you’ve provided, there are a couple of logical fallacies at play:

  1. Ad Hominem: This is an attack directed towards the person making the argument rather than the argument itself. In this case, it’s implied that because the speaker has experience as a former home affairs minister, they should know that things aren’t simple. The phrase suggests that their lack of understanding (or perhaps misunderstanding) is due to their previous role, rather than addressing the substance of the argument.

  2. Appeal to Complexity: This fallacy occurs when someone uses complexity as an excuse for not thinking deeply about an issue or for dismissing evidence or reasoning that runs counter to one’s own position. In this context, “it is never that simple” could be seen as a way of dismissing the initial argument without actually addressing its merits.

While there isn’t an overt appeal to emotion in the phrase itself, the use of fallacies like Ad Hominem and Appeal to Complexity can sometimes subtly appeal to emotions like disdain or frustration with overly simplified arguments. However, these aren’t direct appeals to emotion in the same way that a statement like “You should agree because you’ll feel bad if you don’t” would be.

So, while there are indeed logical fallacies present in this phrase, they’re not explicitly emotional appeals.

Gemma 2 - 9b

Gemma 2 logo

Model: gemma2:9b-instruct-q8_0

Response:

The phrase “Look, on first blush, it all sounds perfectly reasonable: too many people, not enough houses. But it is never that simple, as a former home affairs minister should know” contains a few logical fallacies:

  • Appeal to Authority: The phrase relies on the perceived authority of a “former home affairs minister” to dismiss the initial argument. The speaker assumes that because the person held a position of power, their opinion automatically holds more weight. This fallacy ignores the possibility that the minister might be wrong or that their expertise doesn’t apply to this specific situation.

  • Pooh-Pooh: The phrase “it is never that simple” dismisses the initial argument without providing any real counter-arguments or evidence. It’s a dismissive statement that aims to shut down the conversation rather than engage with the ideas presented.

Let me know if you’d like to explore other examples of logical fallacies!

The phrase “Look, on first blush, it all sounds perfectly reasonable: too many people, not enough houses. But it is never that simple, as a former home affairs minister should know” contains a few logical fallacies:

  • Appeal to Authority: The phrase implicitly suggests that because the speaker mentions a “former home affairs minister,” their opinion holds more weight. This assumes that being a former minister automatically grants expertise on this particular issue.

  • Pooh-Pooh: The phrase “it is never that simple” could be considered a form of pooh-poohing the initial argument. It dismisses the straightforward reasoning without providing substantial counter-arguments or exploring the complexities.

Let me know if you’d like to analyze other phrases or arguments for fallacies!

Gemma 2 - 27b

Model: gemma2:27b-instruct-q3_K_L

Response:

The phrase “But it is never that simple, as a former home affairs minister should know” employs the appeal to authority fallacy.

Here’s why:

  • It assumes expertise: The statement implies that because the person is a former home affairs minister, they automatically have special knowledge about complex social issues like housing shortages.
  • Expertise isn’t always relevant: While a former minister might have some insight into housing policy, their experience doesn’t necessarily make them an expert on all the contributing factors to a housing shortage.

The phrase attempts to dismiss a potentially valid argument (“too many people, not enough houses”) simply because it comes from someone who is perceived as lacking expertise in the field.

Let me know if you have any other phrases you’d like analyzed!

The phrase contains a couple of elements that resemble logical fallacies, although it’s not a straightforward case:

  • Appeal to Authority (Weak): “as a former home affairs minister should know.” This attempts to lend weight to the argument by invoking the speaker’s previous authority on the subject. However, it’s weak because it doesn’t provide any specific expertise or reasoning from the minister to back up their stance.

  • Ad Hominem (Potential): Depending on context and tone, the phrase could be seen as an ad hominem attack if it’s intended to discredit the opposing argument simply based on who is making it rather than addressing the actual content of the argument. For example, if the statement implies the person is incapable of understanding housing policy because they were a minister, that would be an ad hominem.

Important Note: Without further context about what the person actually argued and the overall tone, it’s hard to definitively classify these as strong examples of fallacies.

Let me know if you have any other phrases you’d like me to analyze!

Qwen 2 - 7b

Qwen 2 logo

Model: qwen2:7b-instruct-q8_0

Response:

The phrase you provided contains several logical fallacies:

  1. Appeal to Emotions (Argumentum ad Passiones): The statement “Look, on first blush, it all sounds perfectly reasonable” attempts to evoke a sense of reasonableness and simplicity in the listener’s mind without providing substantial evidence or argumentation.
  2. Red Herring Fallacy: By mentioning that “a former home affairs minister should know,” the speaker introduces an irrelevant point (the knowledge of a specific individual) to divert attention from the main issue at hand, potentially misleading the audience into thinking that the person’s expertise is relevant when it might not be.
  3. Appeal to Authority Fallacy: Although not explicitly stated in this phrase, if someone were to claim that “a former home affairs minister” knows something because they are a former minister, they would be committing an appeal to authority fallacy, assuming their position grants them expertise on the matter without providing evidence for their specific knowledge or argument.

These logical fallacies can undermine the strength of an argument by appealing to emotions rather than logic and relevance.

The phrase you provided:

“Look, on first blush, it all sounds perfectly reasonable: too many people, not enough houses. But it is never that simple, as a former home affairs minister should know.”

does not contain any of the Appeal to Emotions (Appealum ad passiones) fallacies mentioned in your list. However, some potential logical fallacies could be:

  • Ad Hominem if this quote is used as an attack against a specific individual (‘former home affairs minister’) instead of addressing their argument.
  • Red Herring Fallacy: There might be an attempt to divert attention away from the original issue or arguments by mentioning irrelevant topics, although no obvious red herring seems present in the sentence you provided.

There are also other fallacies not mentioned under ‘Appeal To Emotions’ such as:

  • False Dilemma: Presenting a complex issue in simple binary terms where it can’t be adequately discussed. The sentence provided does not imply this kind of fallacy.
  • Appeal to Authority: Using the opinion or identity of someone famous instead of valid evidence to back an argument. Not applicable in your quote since no authority is invoked.

Please clarify if there was another specific phrase you meant for analysis under appeal to emotions, and we would be glad to help with that.

Qwen 2 - 72b

I know, that’s overweight is not from our category, but just curious…

Model: qwen2:72b-instruct-q2_K

Response:

The phrase provided doesn’t directly contain an appeal to emotions as defined in the logical fallacies list. However, it does imply a potential logical fallacy known as “Appeal to authority” or “Argument from authority.” This is because the speaker suggests that the former home affairs minister should know better, implying that their authority on the subject makes their opinion more valid.

The phrase doesn’t directly use an emotional appeal but rather uses the authority of the former home affairs minister’s presumed knowledge as a way to undermine the simplicity of the argument being presented. This can be seen in the structure of the sentence: “But it is never that simple, as a former home affairs minister should know.”

In summary:

  • Logical fallacy present: Appeal to authority
  • Definition: An appeal to authority occurs when an assertion is deemed true based on the credibility or expertise of the person making the claim rather than the strength of the evidence.
  • Example in phrase: “as a former home affairs minister should know” implies that the opinion of someone with such experience carries more weight and could be used to dismiss simpler explanations.

The phrase you’ve provided does not necessarily evoke a specific emotional response; rather, it appeals to the listener’s knowledge and experience by suggesting that simple explanations often don’t tell the full story. However, if we were to analyze it for potential emotional appeal:

Appeal to Authority: It indirectly references “a former home affairs minister” who should know better than accept simplistic explanations. This could be seen as an indirect attempt to lend authority and credibility to the complexity of the issue, which can subtly appeal to our desire for expert guidance.

The phrase does not directly use emotional appeals such as Fear, Guilt, Ridicule, Anger, Spite, Disgust, Flattery, Pity, Patriotism or Pooh-Pooh, but it indirectly references authority (appealing to our respect and deference) to underscore the complexity of issues that are not simply resolved by blaming “too many people” for a lack of houses. This can also be related to:

Appeal to Reason: It prompts us to look beyond simple solutions and consider that problems may have deeper or more complex causes than they appear at first glance.

Overall, while there might not be a strong emotional appeal evident in the phrase, it does attempt to leverage authority and reason - aspects often respected within rational decision-making. This highlights a possible strategic play on emotions of trust towards authoritative figures to instill doubt in simplistic views and encourages more thoughtful considerations by listeners who appreciate complexity in societal issues. A nuanced version could potentially invoke elements related to:

  • Fear: Subtly raising concerns about the oversimplification, thus invoking anxiety around unaddressed complexities.
  • Disgust: Implied distaste for over-simplified narratives might be triggered if framed within contexts criticizing superficial or ignorant policy discussions.

However, these are indirect and may depend heavily on contextual nuances in surrounding discussions. Without additional emotional content or specific appeals that are characteristic of defined fallacies (e.g., fear-mongering, overtly invoking guilt), this phrase is primarily centered around invoking reasoning through authority rather than manipulating feelings. This positioning emphasizes credibility, often appreciated when analyzing logical fallacies which can be manipulated emotions without solid arguments.

To summarize:

  • The statement alludes to authority (“a former home affairs minister”), potentially sparking respect or trust (as it insinuates insider knowledge).
  • It could induce subtle anxiety towards simplism as a threat (though fear tactics aren’t overtly employed here).
  • A more detailed context might be necessary for specific emotional appeal classifications since inherent appeals tend to derive from how listeners relate emotionally with issues beyond the statement itself.

Remember, fallacies typically involve logic errors but often resonate on deeper, emotionally influenced personal biases or societal anxieties – thus making this phrase’s implications worth unpacking carefully around what listeners could subconsciously connect within broader dialogues.