How a forum post becomes data

This site makes claims about what Nigerian farmers discuss. That is only worth anything if you can see how a page of forum argument turned into a row in a table — and where, along the way, something could have been lost or invented.

The short version

Nairaland's agriculture section was walked to build a list of every thread. Each thread's pages were fetched slowly and stored exactly as received, and never fetched again — every later step reads the archive, not the site. Each stored page was parsed into its posts, then read by a language model running on a machine in this house, which was asked to describe the page using a fixed vocabulary agreed in advance. Everything it returned was then checked mechanically against the page it had just read, and anything it could not support was deleted before the record was written.

Why each step is shaped the way it is

The archive is written once and never refetched.

The parser was rewritten many times; the site was asked for each page exactly once. That is both a politeness measure and a reproducibility one — every number here can be recomputed from disk without touching Nairaland again.

The vocabulary was fixed before the full run, not after.

It was derived by reading a stratified sample of 276 pages, then frozen. Choosing labels after seeing all the data lets you fit the vocabulary to the answer you wanted. The cost of freezing is that anything it fails to describe lands in unclassified — which is why that bucket is published rather than hidden.

The vocabulary is enforced during generation, not checked afterwards.

The list of permitted labels is compiled into the model's decoding constraints, so emitting an invented label is structurally impossible rather than merely discouraged. The verifier still checks, because defence in depth is free.

Every claim must name the post it came from.

A theme without a post id is deleted, not displayed. This is the single measure that does most to stop a confident-sounding summary of a page nobody wrote.

Advertising is classified, not discarded.

Most of the corpus is people selling things. Dropping it would have thrown away the price data, which is among the most useful material here. It is labelled instead, so it can be excluded from discussion findings and included in price ones.

A worked example

One real page, followed end to end. Chosen by a rule rather than hand-picked: the first page, in id order, that is a discussion, has between 3 and 14 posts, a verified quote, a price, a pain point citing a post — and a quote the verifier rejected, because an example that only shows success demonstrates nothing about whether the checks do anything.

Rate My Farm Produce Guys · page 8 · 11 posts · nairaland:8198084:8 · read it on Nairaland ↗

1 — What people wrote

5 further posts on the page.

2 — What the model was given

The same posts, rendered by the same function the pipeline uses. Each post is prefixed with its id, because the model must cite ids and anything else that looks like a number invites it to cite that instead.

THREAD: Rate My Farm Produce Guys
PAGE: 8 (11 posts)
VIEWS: 33567
SIGNALS: 7 distinct authors; busiest author wrote 27% of posts; 0/11 posts contain a phone number; 0/11 posts have no text of their own; 5 total likes on the page

post_id=131754948 | budaatum | 2024-08-29 | 2 likes | replying to #131751528
Technically, I was not helping him. I was expressing my gratitude for him helping himself.

post_id=131756254 | Funkybabee | 2024-08-29 | replying to #131749098
Hmmm

22k

post_id=131844100 | LastGiantAlive | 2024-09-04 | replying to #131731281
3,163 to 5,298

post_id=131844218 | ActiveFarms | 2024-09-04 | replying to #131844100
Very far

post_id=131873786 | giftiy | 2024-09-05 | replying to #131731281
56,000 Cobs

post_id=131898503 | LastGiantAlive | 2024-09-07 | replying to #131749098
Nobody won?

post_id=131916856 | ActiveFarms | 2024-09-09 | 1 likes | replying to #131898503
There are winners. Problem is time. I will make time to address when done with farm works

post_id=131920483 | LastGiantAlive | 2024-09-09 | replying to #131916856
Alright chief

post_id=132198100 | frankputer | 2024-09-28 | 2 likes | replying to #131916856
INEC this election no get Result ni.
My mind fully Dey there.

post_id=132198606 | ActiveFarms | 2024-09-28 | replying to #132198100
There was moisture that affected the maize during storage. Drying and packing and all has overwhelmed me. I will get …

3 — What it returned

{
  "unit_type": "discussion_qa",
  "summary": "This thread is a Q&A discussion where farmers share specific maize prices and discuss the challenges of post-harvest storage, specifically moisture damage. The conversation is driven by users asking for rates and the original poster explaining delays in responding due to farm workload.",
  "themes": [
    {
      "label": "market_access_offtakers",
      "evidence_post_id": 131844100
    },
    {
      "label": "storage_post_harvest",
      "evidence_post_id": 132200864
    }
  ],
  "pain_points": [
    {
      "label": "storage_loss",
      "evidence_post_id": 132200864
    }
  ],
  "actor_roles": [
    "smallholder",
    "trainer_consultant"
  ],
  "commodities": [
    "maize"
  ],
  "locations_mentioned": [],
  "coordination_attempt": false,
  "price_signals": [
    {
      "commodity": "maize",
      "price": 6000.0,
      "currency": "NGN",
      "quantity": null,
      "unit": null,
      "location": "Nigeria",
      "post_id": 132200864
    },
    {
      "commodity": "maize",
      "price": 7000.0,
      "currency": "NGN",
      "quantity": null,
      "unit": null,
      "location": "Nigeria",
      "post_id": 132200864
    },
    {
      "commodity": "maize",
      "price": 3163.0,
      "currency": "NGN",
      "quantity": null,
      "unit": null,
      "location": "Nigeria",
      "post_id": 131844100
    },
    {
      "commodity": "maize",
      "price": 5298.0,
      "currency": "NGN",
      "quantity": null,
      "unit": null,
      "location": "Nigeria",
      "post_id": 131844100
    },
    {
      "commodi…

4 — What the verifier refused

Each quote is checked character-for-character against the specific post it names. This one did not match, so it is stored as failed and is never shown as evidence anywhere on this site:

Two further checks run at the same point: a label citing a post that is not on the page is deleted, and a label outside the frozen vocabulary is deleted. Corpus-wide those are 62 and 0 respectively.

5 — What was kept

That record is what every table on this site is counted from. Nothing on a findings page is derived from anything the model said that was not first checked this way.

And the other 58%

Most of this corpus is advertising, and the right answer for an advert is usually nothing. Getting the model to return empty lists rather than inventing themes to fill a field took three attempts to get working — an early version fabricated post ids on precisely these pages. Here it is doing it correctly:

How well it worked

Prose can only be judged by reading it. These are the checks that run mechanically over every page, recomputed live from the records on disk.

Model: Qwen3.5-9B-exl3-5.00bpw · prompt v4 · 8479 pages across 5873 threads.

Quotes that failed verification

Either fabricated, or real words attributed to the wrong post. Both make the permalink wrong, so both fail and neither is shown as evidence anywhere else on this site.