Non-GPU ML/DL AI-server using FreeBSD

dnb

dnb

Some advocacy is needed to use FreeBSD as an AI server system. I'm going to use FreeBSD as the base system for the AI server. Perhaps someone (from the Linux world) will have questions for me about this because of their inertia of thinking, and I will have to answer them something. Here are my arguments briefly:
  • Although it is common practice to use GPUs for AI tasks, there are also serious problems with their use.
  • High cost of GPU in addition to the main system.
  • Significant GPU overhead. For example, the use of modern GPUs places great demands on power supplies. I could use the excellent Supermicro (Ablecom) PWS-865-PQ 865W power supply with any modern processor. But if I need a GPU, then I'll have to buy a more expensive (but not necessarily more reliable) 8pin+8pin power supply.
  • Lack of software for relevant calculations. NVIDIA CUDA/AMD ROCM does not support BSD operating systems.
  • Imagine how wonderful it would be if we decided to abandon graphics accelerators (and their toxic proprietary drivers). We could use FreeBSD (and it would have no competition in the ML/DL space). We could also use a great system like DragonflyBSD. It would be technologically beautiful.
  • There are convincing results of successful use of FreeBSD and DragonflyBSD with AMD EPYC CPU. https://www.phoronix.com/review/bsd-epyc-milan
  • FreeBSD is a system that compiles with LLVM/Clang. The LLVM compiler framework is ubiquitous in the ML/DL field, as systems like Tensorflow/PyTorch perform lowering into the MLIR intermediate representation. This IR supports many ML related dialects, in particular Tensor Operator Set Architecture (TOSA) Dialect https://mlir.llvm.org/docs/Dialects/TOSA/
  • “The big elephant in the room that usually goes unnoticed” when talking about FreeBSD is that FreeBSD compiles entirely using LLVM, while Linux and DragonflyBSD are far from this (and this is where Linux lags more technologically).
  • But all graphic compilers are based on the use of LLVM. Thus, FreeBSD is closer in software to them than other systems, which are a kind of zoo of software that is conceptually unrelated.
  • In recent years, AMD's CPU development has been very powerful. For example, you can easily use a system with 64 or 128 cores. This could be a Supermicro h12ssl or ASROCK romed8 motherboard with an EPYC 7763 processor (or Supermicro MBD-H12DSI-N6-B for two CPUs). Personally I use H12SSL with EPYC 7763 64c/128t processor and it works great with FreeBSD, full support. I didn't even try to install Linux on such systems.
  • Recently, reports have appeared on the development of an EPYC (Turin) processor with 256c/512t. How many more cores does a processor need to eliminate the need for a GPU for most modern AI tasks? I suspect that not all people who use NVIDIA DGX really need such powerful systems.
  • FreeBSD supports Tensorflow/PyTorch in principle. Imagine how great it would be if, having such a processor with 192 cores, you could install TensorFlow in a FreeBSD jail and run your own neural networks.
  • Now we don't have to use NVIDIA, which denied FreeBSD the right to CUDA. https://github.com/NVIDIA/open-gpu-kernel-modules/discussions/244
  • However, Linux support is far from perfect too (https://github.com/ROCm/ROCm/issues). For example, AMD Rocm does not support all GPUs (only the most expensive and energy-consuming ones), and even with them there are many ambiguities if you step one step to the right or left of their instructions. For example, if you are using unsupported Ubuntu 22.04.1 instead of Ubuntu 22.04.2 (https://rocm.docs.amd.com/en/docs-5.5.1/release/gpu_os_support.html). Or if you are using Debian, which is also not supported.

Q: Have you had any experience using FreeBSD for ML? If you have an opinion on this or if you have your own thoughts on using FreeBSD for an ML/DL server, continue my list.

P.S.: seeing the high level of FreeBSD performance (13 and 14), I'm sure that those who created such a productive OS meant it for high-performance applications.
 
Corporations who are serious about AI usually have deep pockets, meaning they will just go with the flow - so most likely Linux. So having no CUDA on FreeBSD will be a major show stopper, because most AI stuff is done on Nvidia's stuff right now.
 
I'm trying to understand your original post. It seems to say the following:
  • You want to do AI calculations, using TensorFlow or PyTorch.
  • The amount of AI calculations you want to do is small enough that it doesn't justify buying a whole GPU or similar accelerator card. Part of the reason is that buying a consumer GPU is expensive is that it needs upgrading the power supply, which would be economically justified if one actually used the whole GPU, but is not for partial use.
  • Therefore you conclude (reasonably) that in your particular situation it is more cost effective to do your calculations on the main CPU's extra cores. You observe (correctly) that modern CPUs have lots of cores, in particular recent AMD offerings. You ignore that core count alone doesn't measure the performance of a CPU, and in particular is not terribly correlated with cost per unit performance.
  • You want to use BSD because you consider it "technologically beautiful" (which is unmeasurable and has no bearing on cost and performance). You claim that "Linux lags technologically", based on the observation that it uses more than one compiler. You consider NVidia's licenses to be "toxic". These are emotional or political arguments, not technical or economic ones.
  • You don't want to even try using the common tool chain, which is Linux based. So while you talk about wanting things to be high performance or inexpensive, you then erect a barrier that prevents you from using the ready-to-use solutions.
A quick search here on this forum will find threads where people have compiled TensorFlow on FreeBSD. So it seems to be possible. Whether it is performant or economically reasonable, I don't know.
 
What is the status of running (not building) Linux CUDA apps on FreeBSD? I have seen mumblings of success, but I haven't tested it myself.
 
  • Thanks
Reactions: dnb
ralphbsz said:
I'm trying to understand your original post. It seems to say the following:
  • You want to do AI calculations, using TensorFlow or PyTorch.
  • The amount of AI calculations you want to do is small enough that it doesn't justify buying a whole GPU or similar accelerator card. Part of the reason is that buying a consumer GPU is expensive is that it needs upgrading the power supply, which would be economically justified if one actually used the whole GPU, but is not for partial use.
  • Therefore you conclude (reasonably) that in your particular situation it is more cost effective to do your calculations on the main CPU's extra cores. You observe (correctly) that modern CPUs have lots of cores, in particular recent AMD offerings. You ignore that core count alone doesn't measure the performance of a CPU, and in particular is not terribly correlated with cost per unit performance.
  • You want to use BSD because you consider it "technologically beautiful" (which is unmeasurable and has no bearing on cost and performance). You claim that "Linux lags technologically", based on the observation that it uses more than one compiler. You consider NVidia's licenses to be "toxic". These are emotional or political arguments, not technical or economic ones.
  • You don't want to even try using the common tool chain, which is Linux based. So while you talk about wanting things to be high performance or inexpensive, you then erect a barrier that prevents you from using the ready-to-use solutions.
A quick search here on this forum will find threads where people have compiled TensorFlow on FreeBSD. So it seems to be possible. Whether it is performant or economically reasonable, I don't know.
Click to expand...
You said everything correctly, but I asked for arguments in favor of the FreeBSD AI-server. Well, for example, "soon we will be able to run AMD NPUs", when we rewrite xdna-driver for FreeBSD: https://github.com/amd/xdna-driver
 
cracauer@ said:
What is the status of running (not building) Linux CUDA apps on FreeBSD? I have seen mumblings of success, but I haven't tested it myself.
Click to expand...
I read here that someone got Blender running successfully. Also, a related issue is NVIDIA GPU forwarding to Bhyve. Some have had success here, although a developer at the conference says that's not the case yet:
forums.freebsd.org

bhyve - Current state of bhyve Nvidia passthrough?

From what I can see the current state is Nvidia GPU passthrough is working with patches to at least Linux and also Windows in some cases, probably with patches against 13.1. Has this changed? Before I start messing with 13.1 with patches, 13.2, or 14.0 or looking at source code trees I thought...
forums.freebsd.org forums.freebsd.org

At least forwarding the GPU to Bhyve would allow us not to have to leave FreeBSD in order to launch Ubuntu LTS to work with NVIDIA.

In any case, even if all my attempts are unsuccessful, there is always the magic word "cross-compilation", right? It’s just that not everyone knows how to cook it correctly.
 
You must log in or register to reply here.
Back
Top