Planning storage server(s)

C

Cromulent

I'm looking into how I'm going to store media files (audio and video). I need to start off small but be able to scale. My plan was to start with say 2TB of NVMe storage and use RAID-Z. I'd also copy the data to a bulk storage server which has slower I/O.

I'd want to be able to add more ZFS storage servers but I'm unsure of the best way to go about this. Unfortunately I don't have any exact figures on the amount of storage needed as it depends entirely on how well the website is received by users and how much data is uploaded.

Is there some way in which I can have multiple servers with ZFS arrays appear as one physical server with one array to the application and then have things load balanced over the multiple storage servers? I could then easily get slow bulk servers as a backup.
 
Scale to what? Using what service. At first I'm like, just add some 22TB disks to scale but I saw "website" and now everything matters, including what exactly the media is you're storing.
 
msplsh said:
Scale to what? Using what service. At first I'm like, just add some 22TB disks to scale but I saw "website" and now everything matters, including what exactly the media is you're storing.
Click to expand...
Very much a site like YouTube but on a tiny size compared to them. It'll be a place to upload media files and then discuss them. The media will be H.264 video using FLAC as the audio codec and then FLAC or ALAC as the audio only codecs.

The website is written in Python and uses the Flask framework. Database is PostgreSQL. ffmpeg will deal with media conversions. Nginx as the HTTP/HTTPS server. Also learning Redis as it could be useful. React for the frontend.

If you want more info let me know.
 
I think the most important number one would need to plan this is the expected feeds and speeds. How many MB/s of video service do you expect? How much upload? What are the file sizes (serving 3KB files is very different from 10GB ones)? What is the serving/upload protocol (looks to me like http(s) from your description)?
 
ralphbsz said:
I think the most important number one would need to plan this is the expected feeds and speeds. How many MB/s of video service do you expect? How much upload? What are the file sizes (serving 3KB files is very different from 10GB ones)? What is the serving/upload protocol (looks to me like http(s) from your description)?
Click to expand...
The one feature I really want to implement is live video streaming using RTMP and Nginx. I'd have the live stream and then record a copy so people could watch it again if they wanted to. In this case it is a "how long is a piece of string" because live video streams might be 5 minutes or an hour.

The main focus is on audio rather than video so things like that will likely move in that direction. File sizes might be 20MB to 40MB for a 5 minute song (rough guess). Video will vary. I only want to start with H.264 because every browser supports it. As soon as I can I'd like to move to AV1.

I can't really estimate MB/sec of video service. But I would say in the low 10 - 20 range of concurrent streams.

Upload protocol is HTTPS.
 
Hm. If you design your system to able to serve the media files directly out of a S3 bucket, then you'll be able to scale to whatever. ZFS array wouldn't matter because you could just mirror it and load balance it in the HTML.
 
msplsh said:
Hm. If you design your system to able to serve the media files directly out of a S3 bucket, then you'll be able to scale to whatever. ZFS array wouldn't matter because you could just mirror it and load balance it in the HTML.
Click to expand...
S3 was my first choice but AWS gets expensive fast especially if you are transferring data into and out of their network in large amounts.
 
You will have great difficulty predicting both the volume of data and the throughput requirements. I would recommend to get a case with proper hotswap frames and start out with a raidz2 or raidz3, now that expansion is possible.

That saves you from the trouble of having to add machines instead of disks.
 
OP,
I use the XigmaNAS system which is constructed on FreeBSD.
Lessons learned: you will quickly outgrow your anticipated need. By a whole lot.
My NAS is for movie storage.

I had six WD Red Pro 4tb drives in stock at the time, and decided to use those.
Under RAIDZ-2, this gives 16tb of usable space.
It is now full.

If you can swing it, start with large disks.
I picked up three 18tb disks, one WD and two Ironwolf for a RAIDZ-1 project.
My NAS are powered up only on demand, so they do not spin on a 24x7 basis.

ZFS allows you to replace small disks with larger ones, one at a time.
After each re-silvering, you do the next disk.
This is expense, far more than buying big disks at the onset.
 
Cromulent said:
S3 was my first choice but AWS gets expensive fast especially if you are transferring data into and out of their network in large amounts.
Click to expand...
S3 is the granddaddy of object storage. I have no experience with open-source alternatives, but that's where I would look first.

Unless I decided to dust off the version I wrote in the early 2000s that used an Oracle database for distributed locking.
 
Cromulent said:
S3 was my first choice but AWS gets expensive fast especially if you are transferring data into and out of their network in large amounts.
Click to expand...
Slap cloudfront in front of it. The point is that if you design your architecture to be "compatible", scaling in the future will be easy.
 
msplsh said:
Slap cloudfront in front of it.
Click to expand...
or bunny.net. They even have stream/video libraries and have very reasonable pricing (0.005$/GB per global zone) as well as fully GDPR-conformant solutions (i.e. EU-originating traffic never leaves the EU; and because they are also an EU-based company this solution is fully GDPR-compliant).
I never used their storage or streaming services, only DNS and some (small/personal use) CDN caching.


Regarding that storage server:
If this should be easily extensible don't use raidz but mirrors. raidz vdevs are carved in stone and can't be changed (except replacing all disks with larger ones), so you have to add identical raidz vdevs to expand the pool. you *could* add any vdev variation, but pool performance will be unpredictable at best...
mirrors also give you by far the best IOPS-performance as well as throughput, compared to if you were to put all disks in one vdev instead of multiple mirrored vdevs.

I wouldn't bother about virtual/distributed filesystems - in 'smaller' scales the (huge) overhead and complexity usually never makes up for the advertised benefits. If you are using a CDN in front of the server(s) anyways, just use one big ZFS pool. You can always add other origin servers to a CDN afterwards, if one day a single server won't suffice.
 
In general, I like the idea of dumping the question of how to serve the audio/video streams on a commercial CDN. Because then you don't have to estimate and plan how much bandwidth you'll need. Obviously, you'll pay $$ for that, and that's a tradeoff.

If you decide to do the serving yourself ...

Cromulent said:
File sizes might be 20MB to 40MB for a 5 minute song (rough guess). ...
But I would say in the low 10 - 20 range of concurrent streams.
Click to expand...
If I take those number literally, I find that every audio stream runs about 0.13 MB/s (that's 40MB divided by 5 minutes), and 20 of them in parallel is 2.7 MB/s. How many disk drives do we need for that? A typical disk drive can do around 200 MB/s in hardware for large sequential single-stream reads, about 100 MB/s for multiple streams, and after file system overhead you can pretty much rely on getting 25-50 MB/s out of it. So for the audio workload alone, you need 1/10th of a disk from a bandwidth point of view.

Video will vary. ... I can't really estimate MB/sec of video service.
Click to expand...
And here is the problem. Let me pluck a number out of the air and say that video bandwidth will be 10x larger than audio (plausible guess). Then given the math above that shows that audio alone needs 1/10th of a disk, you'll need 1 disk for the video bandwidth. The problem is that all these guesses are inaccurate by factors of 2 or 3, and we are stacking several inaccurate estimates on top of each other. So with video you might need perhaps 1/9th of a disk or 9 disks, and that makes a BIG difference, not just financially, but also how you set up the file system.

I have one other suggestion: Don't just blindly by the largest disks you can afford. They are considerably more expensive than smaller disks. For what one 22 TB disk costs, you can get several used 4 TB disks, or several new 4 TB SSDs. The latter will give you more or much more bandwidth. Your disk size will be determined by the balance between capacity need and bandwidth need. So you need to figure out the capacity needed and the bandwidth needed, and then compare the ratio between the two to what SSDs, small disks and large disks give you. But to do any of that you need to first come up with plausible and reasonably accurate estimates.

My advice would be quite different: If you decide to do the serving yourself, then start by finding (perhaps in your parts bin) an old generic case, and three medium-size disk drives (perhaps 4 TB). Set them up as a three-way ZFS mirror (because you need redundancy, and with a three-way mirror you get 3x the read bandwidth compared to write bandwidth, which will fit your workload well). Start running with that setup. After a while, measure capacity and bandwidth usage, and ONLY THEN go buy the next generation hardware. Use actual data and measurement to do the planning.
 
ralphbsz said:
In general, I like the idea of dumping the question of how to serve the audio/video streams on a commercial CDN. Because then you don't have to estimate and plan how much bandwidth you'll need. Obviously, you'll pay $$ for that, and that's a tradeoff.

If you decide to do the serving yourself ...


If I take those number literally, I find that every audio stream runs about 0.13 MB/s (that's 40MB divided by 5 minutes), and 20 of them in parallel is 2.7 MB/s. How many disk drives do we need for that? A typical disk drive can do around 200 MB/s in hardware for large sequential single-stream reads, about 100 MB/s for multiple streams, and after file system overhead you can pretty much rely on getting 25-50 MB/s out of it. So for the audio workload alone, you need 1/10th of a disk from a bandwidth point of view.


And here is the problem. Let me pluck a number out of the air and say that video bandwidth will be 10x larger than audio (plausible guess). Then given the math above that shows that audio alone needs 1/10th of a disk, you'll need 1 disk for the video bandwidth. The problem is that all these guesses are inaccurate by factors of 2 or 3, and we are stacking several inaccurate estimates on top of each other. So with video you might need perhaps 1/9th of a disk or 9 disks, and that makes a BIG difference, not just financially, but also how you set up the file system.

I have one other suggestion: Don't just blindly by the largest disks you can afford. They are considerably more expensive than smaller disks. For what one 22 TB disk costs, you can get several used 4 TB disks, or several new 4 TB SSDs. The latter will give you more or much more bandwidth. Your disk size will be determined by the balance between capacity need and bandwidth need. So you need to figure out the capacity needed and the bandwidth needed, and then compare the ratio between the two to what SSDs, small disks and large disks give you. But to do any of that you need to first come up with plausible and reasonably accurate estimates.

My advice would be quite different: If you decide to do the serving yourself, then start by finding (perhaps in your parts bin) an old generic case, and three medium-size disk drives (perhaps 4 TB). Set them up as a three-way ZFS mirror (because you need redundancy, and with a three-way mirror you get 3x the read bandwidth compared to write bandwidth, which will fit your workload well). Start running with that setup. After a while, measure capacity and bandwidth usage, and ONLY THEN go buy the next generation hardware. Use actual data and measurement to do the planning.
Click to expand...
Thank you very much. That was a very useful post.
 
You must log in or register to reply here.
Back
Top