Thursday, September 28 — Major update and 2 significant news stories

Hey everyone!

I have a big and exciting update to share. It’s longer than usual, so I put it after the news.

Today ChatGPT read 1190 top news stories. After removing previously covered events, there are 2 articles with a significance score over 7.

[7.7] U.S. and Japanese agencies warn companies about China-linked hacker group BlackTech. — Reuters

The U.S. National Security Agency, Federal Bureau of Investigation, and Japanese police have issued a joint advisory warning multinational companies about the China-linked hacker group BlackTech. The advisory urged companies to review their internet routers to minimize the risk of potential attacks from the group, which has been engaging in cyberattacks on governments and tech-sector companies since around 2010. The advisory comes amid heightened tensions between the U.S. and China over cybersecurity issues.

[7.2] Mass exodus of ethnic Armenians from Nagorno-Karabakh after Azerbaijan takeover. — Reuters

More than half of the 120,000 ethnic Armenians in Nagorno-Karabakh have fled the region in less than four days after Azerbaijan took control, marking an unprecedented exodus in the South Caucasus. Azerbaijan has stated that it will respect ethnic Armenian rights, but due to historical tensions and fears of genocide and ethnic cleansing, many Armenians are leaving in fear. As of Thursday morning, over 65,000 people had crossed into neighboring Armenia.

Now, the update.

Adding live coverage and solving topic fatigue.

Three problems bugged News Minimalist website from the start: stale news, bad updates timing, and repetitive content. And I think I finally solved them.

Problems

Since the very beginning of the project, the news was grouped by days. The algorithm collected news for 24 hours, rated all articles, and showed the results for the next 24 hours. The system was built around this newsletter that was sent daily. But it had three major problems.

First, the news was becoming stale. Since the website was static for 24 hours, it considerably lagged behind the events by the end of that window. The problem was even worse than it appeared: since the website didn't change for 24 hours after the update, in the worst-case scenario you could read news with a 48-hour delay!

Second, the website was updated too late for many parts of the world. If you're in the Americas, the website was updated from 5 to 9 a.m. based on your timezone, letting you read fresh news in the morning. But for other parts of the world, the update came too late: around lunch for Europe and dinner in Asia.

The third problem was that deduplication only worked on news within 24 hours. The next day, the deduplication algorithm analyzed the entirely new batch of articles with no overlap. That led to people seeing news about the same event every day, leading to "topic fatigue" — where a single event is covered so much that it's not interesting anymore and causes frustration.

Solution

The first part of the solution is simple: News Minimalist switches to live coverage. Now the website is updated every hour, letting people learn about new events almost in real-time and when they find it convenient.

But the second part took me a long time to figure out.

The deduplication algorithm now works continuously, analyzing a sliding 72-hour window of articles instead of 24-hour non-overlapping batches. This extends the purpose of the algorithm: from finding plain duplicates, to organizing articles into "stories" covering one event. Now, the crucial part here is that stories are sorted by their start time: new stories at the top, and older at the bottom. Once the oldest article in a story falls out of the 72-hour window, it gets replaced by a slightly newer but still old article, keeping the story at the bottom. This should solve the issue of "topic fatigue" — something entirely new and unexpected must happen for an article to break away from the story and form a new story that would appear at the top. The older story will still be available at the bottom and will continue to give ongoing coverage of the event.

This is still a bit raw. I will be adjusting the length of the window and the “closeness” of articles in one story over the next several weeks to improve the feed even more.

I am very excited to share this — this is a new chapter for News Minimalist and I hope you’ll like it: https://www.newsminimalist.com/

Thank you all,
Vadim