_______ __ _______
| | |.---.-..----.| |--..-----..----. | | |.-----..--.--.--..-----.
| || _ || __|| < | -__|| _| | || -__|| | | ||__ --|
|___|___||___._||____||__|__||_____||__| |__|____||_____||________||_____|
on Gopher (inofficial)
URI Visit Hacker News on the Web
COMMENT PAGE FOR:
URI Giving Rust a chance for in-kernel codecs
jmull wrote 6 hours 55 min ago:
This seems like a case of two classic mistakes to me⦠#1 starting
with a solution, and then applying it to problems (imperfectly); #2
solving the problem at the wrong level (related to problem #1) - in
this case at too low a level, which can work, but is a lot more work
than a solution at the right level.
That is: why not sandboxing rather than a rewrite?
saagarjha wrote 6 hours 24 min ago:
Sandboxing an in-kernel codec?
st_goliath wrote 6 hours 15 min ago:
The kernel has API in place to run code in userspace from within a
kernel module ("usermode driver"). That would in theory be one way
something complex in the kernel could be isolated. The obvious
downside being the additional context switches and round trips for
the data adding latency.
Here's a blog post that demonstrates embedding an entire Go program
as a blob into a kernel module and running it in userspace from the
module:
URI [1]: https://www.sigma-star.at/blog/2023/07/embedded-go-prog/
blipvert wrote 6 hours 16 min ago:
eBPF?
jeroenhd wrote 10 hours 18 min ago:
I think these types of drivers, taking care of parsing in a place where
I would personally say parsing should be avoided where possible, is an
excellent place to move towards Rust. The language may not be great at
doing things like low-level memory management, but parsing data and
forwarding it to hardware seems like an excellent use case here.
And yes, you can parse safely in C. Unfortunately, correctness in C has
been proven to be rather difficult to achieve.
andrepd wrote 6 hours 57 min ago:
Why is rust not good at low-level memory management? Can you give an
example?
matu3ba wrote 4 hours 15 min ago:
Hard mode Rust provides the trickyness of separating memory
handling from the (pure) code logic [1] which makes batching inputs
unnecessary hard.
The other main tradeoff is that explicit lifetime annotations make
(big) refactorings or explorative coding harder, unless one uses a
lot copies or shared pointers (unacceptable for Kernel code).
However, I am not familiar how those two tradeoffs ought to be
solved in Kernel.
I would expect that Rust would do rarely changing high level
abstractions expected to never change fundamentally in design
and/or being limited in scope.
URI [1]: https://matklad.github.io/2022/10/06/hard-mode-rust.html
b20000 wrote 11 hours 20 min ago:
Rust must be removed from the kernel. It is a huge time sink for
smaller companies to deal with another language and frameworks. Effort
should be focused on getting more hardware support mainlined such that
device manufacturers have less work which will increase adoption.
Memory safety can be addressed via kernel tools or frameworks and
should not be the job of a language IMHO.
jeroenhd wrote 10 hours 39 min ago:
This reads like a comment from twenty years ago, to be honest.
Memory safety could've been addressed through kernel tools and
frameworks for decades, but it hasn't. And it _should_ be part of the
language, as even low level languages like C try not to clobber
memory and specify undefined behaviour in cases where you may
accidentally end up doing it anyway.
There are good arguments for and against other Rust features such as
the way panic!() works and the strictness of the borrow checker.
However, "C and tooling can do everything your fancy pants new
language does" has been said for longer than I've been alive and yet
every month I see CVE reports about major projects caused by bugs
that would never have passed the Rust compiler.
Out of every reason I can think of, a lack of hardware support seems
like the least likely reason for a hardware manufacturer not to
upstream hardware support. Look at companies like Qualcom, with
massive ranges of devices and working kernel drivers, hacked together
because upstreaming doesn't benefit them. Look at companies like
Apple, who doesn't care if their software works on Linux or not. The
Linux kernel supports everything from 90s supercomputers to drones to
smart toothbrushes, there's no lack of hardware support.
b20000 wrote 29 min ago:
remember java? they forced OO onto users. a huge mistake.
developers should not be forced to use a paradigm. same with memory
safety. if they donât get it, find better developers and pay
more. essentially you are saying rust is great because companies
can hire clueless people and rust will compensate for their
inability.
itâs clear from your comment you never built a device yourself
running linux on an obscure SoC dealing with patches that were
never accepted.
IshKebab wrote 10 hours 39 min ago:
I see these frankly crazy opinions fairly regularly and I'm genuinely
curious how you come to these conclusions.
Have you written much C or C++? What kind of kernel tools or
frameworks are you thinking of? Have you ever used Rust? Are you
familiar with the different kinds of memory errors?
I really struggle to imagine how anyone who is actually familiar with
all this stuff could say things like this but you aren't the first...
b20000 wrote 26 min ago:
i have 20 years of experience writing software in C++.
i donât need or want rust.
hamandcheese wrote 10 hours 45 min ago:
As a user I am not particularly sympathetic to the economic concerns
of some small company if that comes at the expense of my security.
And Rust does seem to be increasing the rate and level of support
small outfits can offer - see Asahi Linux for an example of that.
b20000 wrote 33 min ago:
security or safety is not necessary in many situations and should
not be forced onto users.
if your usage requires that fine. donât place your requirements
on other users.
also you are basically saying adopting linux is only for big tech.
i donât think that is in the spirit of open source.
yjftsjthsd-h wrote 4 min ago:
> security or safety is not necessary in many situations and
should not be forced onto users.
> if your usage requires that fine. donât place your
requirements on other users.
Conversely, if you don't care about code quality feel free to not
use Linux or make your own in-house patches without contaminating
the kernel other people are trying to use.
> also you are basically saying adopting linux is only for big
tech. i donât think that is in the spirit of open source.
If only "big tech" can write kernel code safely then so be it. Of
course, I think that premise is nonsense, but I don't care about
the onramp enough to compromise on security.
phi-go wrote 10 hours 56 min ago:
Isn't Rust only used for kernel modules? So no one needs to depend on
Rust code if not needed?
b20000 wrote 26 min ago:
i donât want to learn rust to write a driver.
yjftsjthsd-h wrote 6 min ago:
Then write your driver in C? The claim is that Rust makes it
easier to produce high-quality drivers, but nobody's mandating
it.
steveklabnik wrote 10 hours 35 min ago:
That is correct, yes.
Faaak wrote 11 hours 12 min ago:
Sorry for this non-hn reply, but "lol"
codedokode wrote 12 hours 4 min ago:
I don't understand why video codecs must be in kernel and run with
supervisor privileges. Why can't they run in a userspace?
saagarjha wrote 6 hours 19 min ago:
Well, for example, if you only have one hardware decoder and two
processes that want to use it someoneâs got to mediate access to it
akira2501 wrote 9 hours 13 min ago:
It's because the value proposition of Rust and the reality of it's
implementation don't match. So, they went through all this effort
to put it in the kernel, but then realized, it's not really useful
for much, outside of making "safe" drivers that no one really needs.
gary_0 wrote 11 hours 42 min ago:
I had the same question. If your chip can do, say, the DCT in
hardware, why not just expose that unit directly to userspace? And if
userspace sends invalid data to that unit, surely the kernel can just
handle the fault and return an error? I must be missing something.
At any rate, it's unfortunate that entire media file formats have to
run in kernel space in order to implement hardware acceleration.
There's no better way to do it?
jeffbee wrote 11 hours 38 min ago:
The kernel has to interpose at least a little bit because some of
these hardware devices can read and write anywhere, so you can't
just let random users sent them commands.
surajrmal wrote 10 hours 23 min ago:
Even without an iommu, there are still benefits to placing this
logic in user space. It doesn't have to be all or nothing.
Ideally if and when it becomes commonplace, iommus will also
become more common.
ok123456 wrote 11 hours 15 min ago:
How is this interposition different from what the virtual memory
subsystem already does? Many other memory-mapped devices have
kernel support. Access to these is usually mediated through udev
or an equivalent on Linux.
singron wrote 11 hours 9 min ago:
I have no specific knowledge, but it seems similar to issues we
had before iommus. E.g. maybe the device accesses physical
memory without memory protection and/or virtual address
mapping, so the kernel needs to do that in its behalf.
saagarjha wrote 6 hours 20 min ago:
This is correct, typically these devices need some page
tables/IOMMU be set up to be told where they should read and
write from.
gary_0 wrote 11 hours 36 min ago:
Why would a hardware codec need the ability to arbitrarily
read/write the entire address space, though? That seems like a
needlessly dangerous design when it could easily have a register
that restricts it to a kernel-designated address range.
Veserv wrote 10 hours 3 min ago:
The general answer is so that you can get the output at a
program-configurable location rather than a hardware-fixed
location. For instance, I want to shove out a frame of data and
then get the decoded data back into a appropriately-sized
static buffer. The hardware could always dump that at memory
address 0x5000 and then I could copy it into my buffer, but
from a programming perspective it is a lot cleaner if it just
shoves the bytes directly into my buffer.
It is analogous to passing a pointer to a buffer to a library
function so that the library can write the contents of the
buffer directly. You rely on the library (device) operating
properly and not writing outside of the designated buffer. That
is the baseline.
As you state, you could add a enforcement mechanism that
defines the extents of memory the external actor is allowed to
store like how many language runtimes check and disallow
out-of-bounds accesses. However, if you have multiple
outstanding buffers, control structures, or other complex
device-accessible structures then enforcing precise "bounds"
checking rapidly demands very complex "bounds"
definition/enforcement. Language runtimes can do this because
they support arbitrary code, but enforcing in hardware that,
say, every access lies within the nodes of a "user-constructed"
red-black tree rapidly becomes infeasible.
You basically get one of two options at that point, either you
rely on the hardware working properly and do nothing, or you
design your driver to only require coarser isolation that fits
within hardware-definable boundaries. Most opt for the former.
If you do the latter then there are various ways of actually
enforcing the isolation such as IOMMUs or you could have a DMA
controller that basically functions as a IO MPU (define ranges
the device can access) (I am not actually aware of any DMA
controllers that actually do this as a security measure, but it
is theoretically possible).
You do have to be careful that they actually enforce the
isolation. For instance, I question if the x86-64 IOMMU
implementation is actually safe against a malicious device due
to certain supported features such as device IO TLBs, but I do
not know about the actual hardware implementation to know for
certain.
binary132 wrote 8 hours 26 min ago:
and even a correct hardware specification could have an
implementation bug...
toast0 wrote 10 hours 3 min ago:
To avoid the cost of copying the data from wherever it was to
where it needs to be...
drdaeman wrote 11 hours 17 min ago:
It shouldnât, but we live in a world where cheapest products
that just barely work overwhelmingly win the market (most
obvious proof is IoT, but it applies to just about everything
else). General consumer market doesnât care about proper
designs, doesnât understand geeky security concerns and
whatever - if something somehow works satisfactorily enough for
acceptable number of situations - it gets released and
marketed, successfully.
Sorry. I hate it too.
gary_0 wrote 11 hours 10 min ago:
Ah. I see. And then the kernel needs to bend over backwards
to cover up the security holes and design flaws. :(
ec109685 wrote 11 hours 59 min ago:
I believe the reason is that itâs not safe to send arbitrary
bitstreams directly to hardware decoders and given the
âstatelessâ nature of them, you need something trusted to run the
full video encoder / decoder logic.
fulafel wrote 11 hours 57 min ago:
What happens if you send bad bitstreams to the hardware?
johntb86 wrote 11 hours 46 min ago:
In theory the hardware should return corrupted video, return an
error, or at least hang, but not anything worse.
It's worse if the data structures specifying memory buffers are
incorrect; then you may be able to read/write arbitrary memory.
Animats wrote 11 hours 41 min ago:
Which hardware decoders are not callable from user space and
unsafe to call from kernel space?
saagarjha wrote 6 hours 21 min ago:
Badly-implemented ones
adastra22 wrote 11 hours 47 min ago:
You can have it overwrite kernel memory.
screcth wrote 11 hours 39 min ago:
Isn't it possible to restrict what memory regions a device may
have access to?
adastra22 wrote 11 hours 36 min ago:
No, it's on the PCIe bus. It sends data straight to RAM,
circumventing the memory controller on the CPU.
SOME systems have write protection logic on the RAM
controllers themselves, but this is not universal.
surajrmal wrote 9 hours 58 min ago:
Iommus are very common on PC grade hardware, as well as
premium smart phones. They just aren't that common on lower
end phones and iot devices, but pcie is also fairly
uncommon on those devices which makes your comment a bit
confusing.
AlotOfReading wrote 9 hours 23 min ago:
There's a lot of moderately powerful embedded Linux
systems out there that either don't have an MMU
equivalent or have one that the vendor BSP doesn't use by
default. Too give one example, Xilinx doesn't set up the
SMMU for AXI DMA devices on Zynq by default, iirc.
Firerouge wrote 11 hours 14 min ago:
> SOME systems have write protection logic on the RAM
controllers themselves
How can one tell if a system has RAM controller based
security, what name does this write protection go by?
stefan_ wrote 10 hours 55 min ago:
Maybe the grandparent is trying to refer to IOMMUs.
adastra22 wrote 8 hours 40 min ago:
This. Your system will almost certainly have an IOMMU.
But that canât be said for all systems that the Linux
kernel supports.
lathiat wrote 3 hours 13 min ago:
However your IOMMU may not actually be in use. It's
not in use by default on Linux and on most Linux
distros as it tends to break things on random
hardware that isn't setup right. It tends to work
most of the time on servers.
Ubuntu 22.04 tried to turn it on by default but
switched it off again due to random mostly graphics
related regressions on random hardware: [1] Some
other history: [2] We really do need it though. I am
always reminded of the very old Apple "Firewire
Memory Bypass" which rendered flames to the screen
just by plugging a firewire device in - because
firewire had direct and originally unprotected DMA
access: [3] It is for this reason that even without
IOMMU, as a workaround, you have to often give
permission to thunderbolt devices to connect. Some
details on that here: [4] There is also a small but
noticable performance hit to using the IOMMU, not so
noticable on a general setup but if you are doing
high-speed disk & network I/O like ceph storage in
excess of 10Gbit/s or millions of IOPS you will
notice it. You can Google that.
You can also run into other weird behaviour, for
example when using kdump to create a kernel crash
dump it will kexec from the old kernel into a new
kernel to produce the crash dump. The system doesn't
go through a firmwire/uefi/bios reset so the hardware
state of network cards, etc, doesn't get reset. So if
you have any hardware driver state that isn't
properly reset, you might for example have your
network card DMA a packet directly into host memory
in the time window before it gets reset. With IOMMU
that might trigger errors, with it off it will
hopefully not overwrite anything important but may
also overwrite something important :)
These things are all of course fixable, but since
it's still off by default much of the time, lots of
these bugs persist for a long time.
Disclaimer: I am not an expert in this area it's just
anecodtes from my life as a Linux Geek & Support
Engineer. Should be about 90% accurate but I am sure
I glossed over some solid details :)
URI [1]: https://bugs.launchpad.net/ubuntu/+source/li...
URI [2]: https://www.phoronix.com/news/Intel-IOMMU-Gf...
URI [3]: https://www.pentestpartners.com/security-blo...
URI [4]: https://wiki.archlinux.org/title/Thunderbolt
Ecoste wrote 11 hours 50 min ago:
Explosions
ben-schaaf wrote 11 hours 59 min ago:
This is for hardware encoding/decoding.
jeffbee wrote 12 hours 8 min ago:
> raw pointer arithmetic and problematic memcpy() calls can be
eliminated, array accesses can be checked at run time, and error paths
can be greatly simplified. Complicated algorithms can be expressed more
succinctly through the use of more modern abstractions such as
iterators, ranges, generics, and the like.
I know why people choose Rust today, but we could have had all of those
benefits decades ago if not for the recalcitrance of reflexive C++
haters.
tialaramex wrote 5 hours 42 min ago:
Crucially, the Rust for Linux people proposed to do the work to make
Rust for Linux. Whenever we see this complaint from C++ people their
idea is basically "LOL, Linus should rewrite kernel to be suitable
for C++" and the answer is No. That's not going to happen.
Once you refuse their fantasy "Don't lift a finger" way to add C++,
they lose interest.
akira2501 wrote 9 hours 11 min ago:
> the recalcitrance of reflexive C++ haters.
I don't reflexively hate C++, just all the implementations of it.
jeroenhd wrote 10 hours 50 min ago:
It took C++ decades to get good, in my opinion. If the kernel team
had switched to C++ decades ago, we probably would've ended up with a
worse kernel code base. For me, C++11 and C++14 created the modern
C++ that's actually usable, but compilers took a while to implement
all those features (efficiently and correctly).
I find the C++ equivalent of the Rust iterators and such to be even
harder to read (almost an accomplishment, given the density of Rust
code); I don't think features like ranges would be expressed more
succinctly using std::range the same way it can be done in Rust, for
instance. I also find C++'s iterators' API design rather verbose, and
I don't think there's much good to be said of implementing generics
through C++ templates. There are good reasons for why the language
was designed this way, but I get the impression succinctness didn't
seem to be a primary objective designing them. Rather, I get the
feeling that the language designers prioritised making the features
accessible to people who already knew C++ and were used to the
complexer side of C++.
b20000 wrote 11 hours 18 min ago:
there is nothing problematic about memcpy, the problem is that you
need to know what you are doing and rust wonât solve that problem.
dale_glass wrote 11 hours 37 min ago:
C++ even today has plenty footguns that will make code unsafe, and
some of the safety comes at performance costs, or uses features that
are ill-suited for the kernel.
pas wrote 11 hours 58 min ago:
Of course, but ergonomics, vibe, or look-and-feel are important. C++
doesn't have a package manager, the .hpp thing is a mess (modules
took ages), SFINAE, preprocessor macros vs proc-macros, and so on.
That said, Linux is a very conservative and idiosyncratic project.
It's really the bazaar. (No issue tracker, firehose of emails, etc.)
flohofwoe wrote 11 hours 42 min ago:
Not to mention: what C++ subset to use, and how to deal with the
endless discussion-circlejerks around that one topic alone.
For instance most of the C++ stdlib is useless for kernel (or
embedded) development, so you end up with a C++ that's not much
more than a "C with namespaces". At least Rust brings a couple of
actual language improvements to the table (not a big fan of Rust,
but if the only other option is C++, then Rust is the clear
winner).
jeffbee wrote 10 hours 59 min ago:
None of the C stdlib can be used in the kernel, either, and yet
the kernel is written in C.
Zircon is an existence proof that you can use the C++ std library
in a kernel. Do they use every damned thing in std? No, but they
do use array, pair, unique_ptr, iterators, and more.
SAI_Peregrinus wrote 8 hours 41 min ago:
> None of the C stdlib can be used in the kernel, either,
Except the bits allowed in a freestanding environment and
(stretching the definition of "used in") the "nolibc" partial C
standard library the kernel includes for environments that
don't have any other libc available (`tools/include/nolibc`,
mostly used for kernel tests where there's no userspace at
all).
jsheard wrote 11 hours 50 min ago:
> (modules took ages)
They're arguably still not there yet, sure the standard was
ratified years ago but the implementations are still a mess and
uptake is almost nonexistent.
vlovich123 wrote 11 hours 52 min ago:
is the kernel using cargo? I donât think the package manager is
relevant.
pengaru wrote 11 hours 26 min ago:
absolutely not
steveklabnik wrote 11 hours 34 min ago:
It is not using Cargo.
DIR <- back to front page