Commit Graph
9 Commits
Author SHA1 Message Date
dualfroz c1d1f5cfdf Describe this repository 2026-10-22 21:02:38 +02:00
dualfroz 06c7822f79 Skip listing containers whose entries hold user cards or quotes
A thread without DiscussionForumPosting markup, where every poster writes
once, still passed the teaser test: each post opens with a linked name that
no other post repeats. On katzen-forum.net with the JSON-LD removed and one
post per poster, recall took the whole container (3,573 chars, seven user
cards instead of one).

Teasers hold neither a description list nor a quote, while forum posts keep
the poster's user card in one and quote each other, so an entry holding
either now rules the container out.
2026-10-10 15:34:44 +02:00
dualfroz 77465a50e7 Keep the listing container off forum threads
Thread pages line up their posts as sibling <article> elements too, so in
recall mode the container took every post along with its user card and
quotes (katzen-forum.net, XenForo).

The container is now used only when every entry is a teaser: its first
heading links to a page of its own, which no other entry links to. Poster
names come back along a thread or carry no link. On top of that it is never
used on pages already detected as thread forums (DiscussionForumPosting).
2026-10-10 15:26:37 +02:00
upstream 03e823a030 Upstream adbar/trafilatura state of the touched files before 1377ec86cc 2026-10-02 17:03:46 +02:00
dualfroz 1884a449e8 Use the listing container only when it widens a short first entry
On correctiv.org the article page preloads the next twelve full stories as
sibling <article> elements, so with favor_recall the listing container took
all of them (98k chars instead of 9.6k). A teaser strip beside a short body
(mopo.de) was also taken over the body.

The container is now used only when the element BODY_XPATH picks on its own
is one of its entries and shorter than the recall sure-thing length
(10x min_extracted_size).
2026-10-10 15:26:09 +02:00
dualfroz 47e47146b7 tests: cover listing pages across settings, precision and single-article pages (#774) 2026-10-10 15:26:09 +02:00
dualfroz ebc0ed3341 Use the listing expression only when its container holds most of the page text 2026-10-10 15:26:09 +02:00
dualfroz ce36d8e544 fix: keep every entry heading on listing pages (#774)
A blog index, archive or category page holds several <article> siblings.
BODY_XPATH's "(.//article)[1]" keeps the first entry only, so the page
looks under-extracted and escalates to justext, which emits paragraphs
and drops every heading.

Give the recall pass a listing expression that takes the container of
the sibling articles, so the retry returns all entries with their heads
and wins the escalation. Recall only: on an article page the same shape
is a related-posts strip, whose container would hide the body.
2026-10-10 15:26:09 +02:00
upstream 7568e2a786 Upstream adbar/trafilatura state of the touched files before 49484caaab 2026-09-11 23:09:08 +02:00