Holllo LogoHolllo

What's new in Fed? Development update 1

A little over a week ago I introduced you to Fed and showed you its capabilities, so today I'd like to share a little about what I've done since then.


Images in feeds will no longer overflow if they're larger than the designated content's width. I noticed this 2 seconds after I added an Atom feed to this blog and looked at it in Fed. :P

The image overflow problem before and after it was fixed.

Some custom logic for Youtube feeds was added so that videos are embedded in the content.

Example of a Youtube embed, the shown video is Yorushika - Prostitution.

I added the Inter and Hasklig fonts (sans-serif and monospace, respectively) to the assets pipeline. These are my 2 favorite fonts and are both open-source, and I like to use them wherever possible. You've been seeing them in use in all the images so far, they're great.

The downloader logging was trimmed a little bit to be less verbose.

Previously it looked something like this
[Downloader] Downloading feed 4303.
[Downloader] Sleeping for 1 second.
[Downloader] Downloading feed 4304.
[Downloader] Sleeping for 1 second.
[Downloader] Skipping 4302 as it was downloaded recently.
[Downloader] Skipping 4301 as it was downloaded recently.
And now it looks like this
[DL] Downloaded feed 5697, sleeping for 1 second.
[DL] Downloaded feed 5698, sleeping for 1 second.
[DL] Skipping 5701 as it was downloaded recently.
[DL] Skipping 5704 as it was downloaded recently.

Downloader Expansion

While in the Introducing Fed post I only briefly touched on the automatic downloader, I intend for it to become a large part of Fed in the future. And with this update I have started the foundational work that is needed to do just that.

Before this update, the downloader was very simple and it essentially worked in 2 steps that loop forever:

  1. Go through and download all the feeds.
  2. Wait until we're on top of the next hour. In short, download when it reaches the next HH:00.

If you were to draw a diagram of what Fed looked like in terms of threads and the downloader's behaviour, it would look like this:

The simple downloader diagram.

When Fed is started, it creates 2 threads. One to run the website* (the rocket thread, named after the Rocket framework Fed uses) and another to run the downloader. And they run independently of each other forever (well, until the program stops, but you get what I mean).

* Note, Rocket automatically multithreads the application, but for our intents and purposes, we just create one thread and then Rocket does whatever it needs to in there.

Now it doesn't take much to realize there isn't much room for expansion here. So, to lay the foundations for expansion, I set myself the challenge of adding 1 simple feature:

I want to be able to click a button on the website that starts the downloader.

Simple, right? Well, it turns out it actually was pretty simple. Let's break it down.

Break It Down

We need to be able to do 2 things:

  1. To send messages from anywhere in Fed, more specifically from any thread.
  2. To receive those messages in the downloader.

And lucky for us, the Rust standard library provides us with something exactly for that.

Introducing mpsc::channel(), "multi-producer single-consumer" channels:

This module provides message-based communication over channels [...] A Sender or SyncSender is used to send data to a Receiver.

Both senders are clone-able (multi-producer) such that many threads can send simultaneously to one receiver (single-consumer).

So with that knowledge, all we have to do to make this work is:

  1. Create a channel.
  2. Give our downloader thread the receiver end.
  3. Give our rocket thread the sender/transmitter end.
  4. Make the downloader wait for an instruction before downloading.
  5. And then do whatever we want with it.

To enable Fed to still have the automated, "on the hour" downloading functionality though, we'll have to create a separate "timer" thread that will then use our newly discovered channel to send an instruction to the downloader, to make it run.

At the same time, we can add a button to the website that does exactly the same, except with a different "force download" instruction. And because these are "multi-producer" channels, this is very easy to do.

Oh Yeah, It's All Coming Together

Let's go back to our little diagram we had and update it with all the things that we've discovered and what it will look like with channels:

The slightly less simple downloader diagram.

The diagram explained

For the purposes of keeping this post relatively simple I've left some specific details out that you can see in the diagram, but I'll go through the important things here if you're interested.

  1. When Fed starts, we create our channel and grab the transmitter and receiver.
  2. Clone the transmitter so we have 2, one for the rocket thread and another for the timer thread.
  3. We create our Rocket server and give it a Mutex-wrapped transmitter. To have Rocket manage this transmitter for us and make it available wherever we want, it needs to be thread-safe and Mutex is an easy way to do that.
  4. We start the timer thread with its transmitter.
  5. We start the downloader thread with its receiver.
  6. And then we start the Rocket thread.

If you'd like to see what this looks like in (my very bad Rust) code, here are the relevant lines.

To specify what exactly the downloader should do, a DownloaderInstruction enum was made that we wait for. And then we can just do an == check whether or not to force download.

Since the instruction enum only has 2 values that are near identical, we don't do much with it yet other than always start the downloader. But this lays the foundation so in the future another value could be added that instructs the downloader to stop, or start over, or something else entirely. The possibilities are endless.

Now that we have one-way communication, all that's left to do is to create the button itself and add a new route that sends the force download instruction. But that's very simple (and I haven't done it yet), so it's for another time.

I hope you found this post somewhat informative, hopefully not too incoherent, entertaining even maybe, and I thank you for sticking through it.

Discuss this post