Hi, everybody!

I just wanted to offer an update concerning our recent downtime, along with some of the video transcoding and storage issues that have affected us over the summer. There are three persistent issues we've had to struggle with over the past few months that have made day-to-day operations a headache.

Namely, these three things:
1. Transcoding Degradation - videos would fail to transcode successfully, and sometimes ended up getting caught in a loop of trying to transcode again, then failing, before starting the whole process over.
2. Storage Transition Failures - Sometimes, the transition between local server storage and Object Storage would fail. In a one-off situation, this isn't too much of a problem, as the video player will still work in most cases. However, large video files would occasionally fill up the disk over time, which would lead to Problem 3.
3. Server Crashes - Occasionally, our server would fail because the disk was so full that the application could no longer keep running.

I'm ecstatic to announce that we've found the root culprit: data chunking. See, when PeerTube moves files from one storage place to another, it needs to ensure that the files being sent over are an acceptable file size. If it's too big, the Object Storage provider might never be able to take the files and then tell the server "okay, I've got it"

It's a weird issue, and the only indications we were getting from the debugging side was the server just silently failing without any info. Now we know what the problem is - and we've managed to process the videos that have been stuck from July 2024 to November 2024. Doing this allowed us to claw back about 120GB of emergency local storage, and the whole process is a lot smoother now.

We apologize for any headaches or frustrations during this time.