Contents
This is a record of a duplicate-content cleanup on marketing365.vn. The duplicates were not between this post and another one — they were inside the same post itself: readers would scroll down, see the exact same section heading they had just read, then read the exact same paragraph again. One post repeated as many as five times.
This article records three things: how I found it, one measurement error that nearly led me to conclude “no posts need to be removed,” and the numbers after the cleanup. It also ends on an unhappy note: three days later, the problem came back.
The first clue: one post was twice as long as normal
I was not looking for this bug. I was reorganizing content by topic and listing posts by length to see which ones were worth splitting in two. The longest post came in at 79,123 characters in the body. The second-longest was over 56,000. By comparison, a normal post on the site sits around 8,000 to 20,000.
Opening the 79,123-character post made the problem obvious: the “Frequently Asked Questions” section appeared five times, each copy identical. Opening the 56,000-character post showed that the second half was exactly the same as the first, copied from the top.
One post can be called an accident. Two in a row, with the same pattern, means the whole site needs to be scanned. The question was how to scan it and get the right number.
The method: split by section headings, then hash each block
The naive approach is to look for identical strings inside a post. That fails immediately, because two “duplicate” passages are often not identical character for character: one has an extra space, another differs by exactly one digit in an image filename.
The workable approach is to split the body at every section heading, normalize each block, then hash it into a short code. Two blocks with the same hash mean the same content, even if they are separated by thirty thousand characters.
# split the body at every section heading (h2 and h3), not just h2
$parts = preg_split('/(?=<h[23][ >])/i', $content);
# each block: strip tags, collapse whitespace, then hash
foreach ($parts as $p) {
$plain = preg_replace('/s+/u', ' ', wp_strip_all_tags($p));
$hash[] = md5(trim($plain));
}
# any hash that appears 2 or more times = duplicated block
Running this across all published posts returned 43 posts with repeated section headings inside themselves. Four of them repeated as full content blocks, while the other 39 were definition posts and had only one thing left over: a second “Frequently Asked Questions” label, with no text underneath.
The hashes of the repeated blocks matched 100%. That means they were truly copied, not rewritten from the same outline — two very different things when you are trying to find the cause.
A one-character offset produced a result of “remove 0 posts”
For the 39 definition posts, I needed to answer one simple question: was there any content under that extra label? If yes, each post had to be read manually. If no, only the label needed to be removed, safely, in bulk.
The first run returned:
WILL REMOVE: 0 posts | skipped: 40
There are two ways to read that number. The pleasant one: every label has content underneath it, so there is nothing to remove and the job is done. The unpleasant one: my script is measuring the wrong thing.
It was the second. I measured the body under the label by cutting it off at the next <h2> tag. But under that extra label there was a series of <h3> tags from another section. The stopping point was pushed too far out, the measured block grew longer, and every post looked like it “had content.” Changing the stopping point to any heading, with the same dataset and the same script, but only one character different in the search pattern:
WILL REMOVE: 40 posts | skipped: 0
This is the most annoying kind of mistake, because it does not throw an error. The command runs smoothly, prints a neat number, and that number reverses the decision. If I had trusted the first run, those 40 posts would still have their empty labels today.
From that point on, I added a habit: whenever a measurement returns exactly 0 or exactly 100%, I treat it as suspicious, not as a result. Too-round numbers are often a sign that the measurement point is in the wrong place.
Before cutting, check the images or you will lose them
With the four full-block duplicates, cutting was much riskier. The copied versions were not clean duplicates: some had internal links inserted, and some used different image files — the same image, but with filenames that differed by exactly one digit at the end, like -89.jpeg and -89-1.jpeg.
That means if you simply keep the first block and cut the later ones, you may also cut the only image that exists in the final block. So the mandatory step before cutting is to list the image filenames in each block, compare them, and only cut the parts that do not contain unique images. Wherever a unique image sits in a block that will be cut, move the image into the block that will be kept first, then cut.
Links were the same story. These four posts were part of the batch where I had previously attached broken auto-generated links — I wrote about that separately in 412 posts were given off-topic links, and the number 479 I reported was wrong. Cutting a block that contains links without checking means the post loses its links, and the next link count will be wrong again.
The numbers after the cleanup
- The post with the FAQ block repeated five times: from 79,123 down to 42,051 characters.
- The post whose second half was identical to the first: from 56,089 down to 30,488.
- The two posts with several repeated sections in the middle: from 53,058 down to 41,068 and from 30,367 down to 26,118.
- The 39 definition posts: only the empty label was removed, with no text lost.
One small detail is worth stating clearly: when I rescan today, those four posts come out at 42,027, 30,365, 41,014 and 26,098 — a difference of a few dozen characters from the numbers recorded in the cleanup log. That is because the posts were edited again after the cleanup, not because the cut was incomplete. I kept both columns in the image instead of only the prettier one.
There is one thing I could not do: trace the root cause. Those four posts were generated by an automated process, and I could not recover the exact run that stitched multiple rounds of content into one post. At the time, I told myself that once the cleanup was done, things would be fine.
Three days later, the problem came back
Today, 14/08/2026, before writing this article, I reran the same script on 1,633 published posts to verify the numbers. The old group of 43 posts was clean, with no repeats. But the list returned a new post.
The post Cách tối ưu internal link cho website, published on 12/08/2026 — three days after the cleanup. It has 22 section headings, 7 of which repeat, a body length of 99,939 characters, and roughly three repeated passes. It is longer than the heaviest post from the previous batch.
I did not cut it myself. The three copies inside the post are not exactly identical — their hashes differ, because one version had internal links inserted while another did not. Which copy to keep and which to cut is a content decision, not a mechanical one, so I reported it back to the person in charge to decide. I am recording it here because that is the real state of the site as of the time this article was scheduled.
Four things I took away from this:
- Measure by hashing each block, not by comparing strings. Two copied blocks are rarely identical character for character.
- The stopping point of the cut matters as much as the cut itself. Shifting from “level-2 heading” to “any heading” shifts the result from 0 posts to 40 posts.
- If the result looks too neat, check the measurement before celebrating.
- If you clean up without finding the cause, you have only cleaned up, not fixed it. Three days later it came back, and this time the post was heavier.
The numbers in this article were measured on marketing365.vn, and the exact date is written next to each number. Anyone who wants to verify them can open the two longest posts and read them — the cut sections are gone, while the 12/08 post is still repeating.
