_______ __ _______
| | |.---.-..----.| |--..-----..----. | | |.-----..--.--.--..-----.
| || _ || __|| < | -__|| _| | || -__|| | | ||__ --|
|___|___||___._||____||__|__||_____||__| |__|____||_____||________||_____|
on Gopher (inofficial)
URI Visit Hacker News on the Web
COMMENT PAGE FOR:
URI What makes Intel Optane stand out (2023)
ritcgab wrote 2 hours 21 min ago:
All those nice numbers are just beaten by the unit cost. And the
ecosystem is a mess.
bluedino wrote 3 hours 3 min ago:
Now do Intel's HBM/CPU Max
myself248 wrote 3 hours 23 min ago:
My kingdom for a MicroSD card with Optane inside. My dashcam wants it
soooo badly.
twotwotwo wrote 3 hours 43 min ago:
One potential application I briefly had hope for was really good power
loss protection in front of a conventional Flash SSD. You only need a
little compared to the overall SSD capacity to be able to correctly
report the write was persisted, and it's always running, so there's
less of a 'will PLP work when we really need it?' question. (Maybe
there's some use as a read cache too? Host RAM's probably better for
that, though.) It's going to be rewritten lots of times, but it's
supposed to be ready for that.
It seems like there's a very small window, commercially, for new
persistent memories. Flash throughput scales really cost-efficiently,
and a lot is already built around dealing with the tens-of-microseconds
latencies (or worse--networked block storage!). Read latencies you can
cache your way out of, and writers can either accept commit latency or
play it a little fast and loose (count a replicated write as safe
enough or...just not be safe). You have to improve on Flash by enough
to make it worth the leap while remaining cheaper than other approaches
to the same problem, and you have to be confident enough in pulling it
off to invest a ton up front. Not easy!
wtallis wrote 3 hours 28 min ago:
> One potential application I briefly had hope for was really good
power loss protection in front of a conventional Flash SSD.
That was never going to work out. Adding an entirely new kind of
memory to your storage stack was never going to be easier or cheaper
than adding a few large capacitors to the drive so it could save the
contents of the DRAM that the SSD still needed whether or not there
was Optane in the picture.
hedora wrote 3 hours 31 min ago:
Any decent SSD has capacitor (enterprise) or battery backed (phones)
DRAM. Therefore, a sync write is just âcopy the data to an I/O
buffer over PCIeâ.
For databases, where you do lots of small scattered writes, and lots
of small overwrites to the tail of the log, modern SSDs coalesce
writes in that buffer, greatly reducing write wear, and allowing the
effective write bandwidth to exceed the media write bandwidth.
These schemes are much less expensive than optane.
olavgg wrote 1 hour 19 min ago:
I have tried multiple enterprise SSD's, for sync writes. Nothing
comes close to Optane Dimm, even Optane NVMe is 10x slower than
PDIMMS.
URI [1]: https://forums.servethehome.com/index.php?threads/so-i-tes...
zozbot234 wrote 3 hours 40 min ago:
> It seems like there's a very small window, commercially, for new
persistent memories. Flash throughput scales really cost-efficiently
Flash is no bueno for write-heavy workloads, and the random-access
R/W performance is meh compared to Optane. MLC and SLC have better
durability and performance, but still very mid.
rkagerer wrote 3 hours 55 min ago:
Did anyone ever see retention issues like this guy reported on one of
his older models?
URI [1]: https://goughlui.com/2024/07/28/tech-flashback-intel-optane-3d...
zozbot234 wrote 3 hours 50 min ago:
That's data retention issues on the very first read-through of the
media after sitting in cold storage for many years, with subsequent
performance returning to normal. It's definitely something to be
aware of (and kudos to the blog poster for running that experiment)
but worn-out NAND will behave a lot worse than that.
exmadscientist wrote 4 hours 9 min ago:
Around the time of Optane's discontinuation, the rumor mill was saying
that the real reason it got the axe was that it couldn't be shrunk any,
so its costs would never go down. Does anyone know if that's true? I
never heard anything solid, but it made a lot of sense given what we
know about Optane's fab process.
And if no shrink was possible, is that because it was (a) possible but
too hard; (b) known blocks to a die shrink; or (c) execs didn't want to
pay to find out?
georgeburdell wrote 3 hours 19 min ago:
Flash has the same shrink problem. And the solution for Optane was
the same: go 3D
exmadscientist wrote 3 hours 1 min ago:
I don't think the shrink problem is at all the same for the two
technologies. There are some really weird materials and production
steps in Optane that are simply not present when making Flash
cells.
hedora wrote 3 hours 25 min ago:
I think it was killed primarily because the DIMM version had a
terrible programming API. There was no way to pin a cache line,
update it and flush, so no existing database buffer pool algorithms
were compatible with it. Some academic work tried to address this,
but I donât know of any products.
The SSD form factor wasnât any faster at writes than NAND +
capacitor-backed power loss protection. The read path was faster,
but only in time to first byte. NAND had comparable / better
throughput. I forget where the cutoff was, but I think it was less
than 4-16KB, which are typical database read sizes.
So, the DIMMs were unprogrammable, and the SSDs had a âsometimes
faster, but it dependsâ performance story.
exmadscientist wrote 3 hours 11 min ago:
The DIMMs were their own shitshow and I don't know how they even
made it as far as they did.
The SSDs were never going to be dominant at straight read or write
workloads, but they were absolutely king of the hill at mixed
workloads because, as you note, time to first byte was so low that
they switched between read and write faster than anything short of
DRAM. This was really, really useful for a lot of workloads, but
benchmarkers rarely bothered to look at this corner... despite it
being, say, the exact workload of an OS boot drive.
For years there was nothing that could touch them in that corner
(OS drive, swap drive, etc) and to this day it's unclear if the
best modern drives still can or can't compete.
myself248 wrote 3 hours 18 min ago:
It sounds like they didn't do a good job of putting the DIMM
version in the hands of folks who'd write the drivers just for fun.
The read path is sort of a wash, but writes are still unequalled.
NAND writes feel like you're mailing a letter to the floating
gate...
zozbot234 wrote 3 hours 15 min ago:
Isn't this addressed by newer PCIe standards? Of course, even the
"new" Optane media reviewed in OP is stuck on PCIe 4.0...
zozbot234 wrote 4 hours 4 min ago:
That's at least physically half-plausible, but it would be a terrible
reason if true. 3.5 in. format hard drives can't be shrunk any, and
their costs are correspondingly high, but they still sell - newer
versions of NVMe even provide support for them. Same for LTO tape
cartridges. Perhaps they expected other persistent-memory
technologies to ultimately do better, but we haven't really seen
this.
Worth noting though that Optane is also power-hungry for writes
compared to NAND. Even when it was current, people noticed this.
It's a blocker for many otherwise-plausible use cases, especially re:
modern large-scale AI where power is a key consideration.
wtallis wrote 3 hours 33 min ago:
> 3.5 in. format hard drives can't be shrunk any,
You're looking at the entirely wrong kind of shrinking. Hard drives
are still (gradually) improving storage density: the physical size
of a byte on a platter does go down over time.
Optane's memory cells had little or no room for shrinking, and
Optane lacked 3D NAND's ability to add more layers with only a
small cost increase.
pgwalsh wrote 4 hours 17 min ago:
Sure, they were expensive but they have great endurance and sustained
read and write speeds. I use one in my car for camera recordings. I had
gone through several other drives but this one has been going on 3 or 4
years now without issue. I have a couple more in use too. It's a shame
this tech is going away because it's excellent.
rkagerer wrote 4 hours 22 min ago:
My understanding is Optane is still unbeaten when it comes to latency.
Has anyone examined its use as an OS volume, compared to today's
leading SSD's? I know the throughput won't be as high, but in my
experience that's not as important to how responsive your machine feels
as latency.
dmayle wrote 1 hour 6 min ago:
I run two 1.5TB Optanes in raid-0 with XFS (I picked them up for $300
each on sale about two years ago). These are limited to PCIE 3.0 x4
(about 4GB/s max each). I also have a 64GB optane drive I use as my
boot drive.
It's hard to tell you, because it's subjective, I don't swap back and
forth between an SSD and the optane drives. I have my old system,
which has a 2TB Samsung 980 Pro NVME drive (PCIE 4.0 x4, or 8GB/s
max) as root, and a Sabrent rocket 4 plus 4TB drive secondary (also
PCIE 4.0), so I ran sysbench on both systems, so I could share the
differences. (Old system 5950X, new system 9950X3D).
It feels snappier, especially when doing compilations...
Sequential reads:
I started with a 150GB fileset, but it was being served by the kernel
cache on my newer system (256GB RAM vs 128GB on the old), so I
switched to use 300GB of data, and the optanes gave me 5000 MiB/s for
sequential read as opposed to 2800 MiB/s for the 980 Pro, and 4340
MiB/s for the Rocket 4 Plus.
Random writes alone (no read workload)
The optane system gets 2184 MiB/s, the 980 Pro gets 32 MiB/s, and the
Rocket 4 Plus gets 53 MiB/s.
Mixed workload (random read/write)
The optanes get 725/483 as opposed to 9/6 for the 980 Pro, and 42/28
for the Rocket 4 Plus.
2x1.5TB Optane Raid0:
Prep time:
`sysbench fileio --file-total-size=150G prepare`
161061273600 bytes written in 50.41 seconds (3047.27 MiB/sec).
Benchmark:
`sysbench fileio --file-total-size=150G --file-test-mode=rndrw
--max-time=60 --max-requests=0 run`
WARNING: --max-time is deprecated, use --time instead
sysbench 1.0.20 (using system LuaJIT 2.1.1741730670)
Running the test with following options:
Number of threads: 1
Initializing random number generator from current time
Extra file open flags: (none)
128 files, 1.1719GiB each
150GiB total file size
Block size 16KiB
Number of IO requests: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Initializing worker threads...
Threads started!
File operations:
reads/s: 46421.95
writes/s: 30947.96
fsyncs/s: 99034.84
Throughput:
read, MiB/s: 725.34
written, MiB/s: 483.56
General statistics:
total time: 60.0005s
total number of events: 10584397
Latency (ms):
min: 0.00
avg: 0.01
max: 1.32
95th percentile: 0.03
sum: 58687.09
Threads fairness:
events (avg/stddev): 10584397.0000/0.00
execution time (avg/stddev): 58.6871/0.00
2TB Nand Samsung 980 Pro:
Prep time:
`sysbench fileio --file-total-size=150G prepare`
161061273600 bytes written in 87.15 seconds (1762.53 MiB/sec).
Benchmark:
`sysbench fileio --file-total-size=150G --file-test-mode=rndrw
--max-time=60 --max-requests=0 run`
WARNING: --max-time is deprecated, use --time instead
sysbench 1.0.20 (using system LuaJIT 2.1.1741730670)
Running the test with following options:
Number of threads: 1
Initializing random number generator from current time
Extra file open flags: (none)
128 files, 1.1719GiB each
150GiB total file size
Block size 16KiB
Number of IO requests: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Initializing worker threads...
Threads started!
File operations:
reads/s: 594.34
writes/s: 396.23
fsyncs/s: 1268.87
Throughput:
read, MiB/s: 9.29
written, MiB/s: 6.19
General statistics:
total time: 60.0662s
total number of events: 135589
Latency (ms):
min: 0.00
avg: 0.44
max: 15.35
95th percentile: 1.73
sum: 59972.76
Threads fairness:
events (avg/stddev): 135589.0000/0.00
execution time (avg/stddev): 59.9728/0.00
4TB Sabrent Rocket 4 Plus:
Prep time:
`sysbench fileio --file-total-size=300G prepare`
322122547200 bytes written in 152.39 seconds (2015.92 MiB/sec).
Benchmark:
`sysbench fileio --file-total-size=300G --file-test-mode=rndrw
--max-time=60 --max-requests=0 run`
WARNING: --max-time is deprecated, use --time instead
sysbench 1.0.20 (using system LuaJIT 2.1.1741730670)
Running the test with following options:
Number of threads: 1
Initializing random number generator from current time
Extra file open flags: (none)
128 files, 2.3438GiB each
300GiB total file size
Block size 16KiB
Number of IO requests: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Initializing worker threads...
Threads started!
File operations:
reads/s: 2690.28
writes/s: 1793.52
fsyncs/s: 5740.92
Throughput:
read, MiB/s: 42.04
written, MiB/s: 28.02
General statistics:
total time: 60.0155s
total number of events: 613520
Latency (ms):
min: 0.00
avg: 0.10
max: 8.22
95th percentile: 0.32
sum: 59887.69
Threads fairness:
events (avg/stddev): 613520.0000/0.00
execution time (avg/stddev): 59.8877/0.00
aaronmdjones wrote 3 hours 35 min ago:
I have a 16 GiB Optane NVMe M.2 drive in my router as a boot drive,
running OpenWRT.
It's so incredibly fast and responsive that the LuCI interface
completely loads the moment I hit enter on the login form.
hamdingers wrote 4 hours 0 min ago:
> Has anyone examined its use as an OS volume, compared to today's
leading SSD's?
Late last year I switched from a 1.5tb Optane 905P to a 4tb WD Blue
SN5000 NVMe drive in a gaming machine and saw improved load times,
which makes sense given the read and write speeds are ~double. No
observable difference otherwise.
I'm sure that's not the use case you were looking for. I could
probably tease out the difference in latency with benchmarks but
that's not how I use the computer.
The 905P is now in service as an SSD cache for a large media server
and that came with a big performance boost but the baseline I'm
comparing to is just spinning drives.
exmadscientist wrote 3 hours 6 min ago:
Unfortunately a gaming machine workload is so read-heavy that I
wouldn't expect Optane to square up well. Gaming is all about read
speed and overall capacity. You need that heavy I/O mix, especially
with low latency deadlines, to see gains from Optane. That limited
target use case, coupled with ignorant benchmarking, always limited
them.
rkagerer wrote 3 hours 46 min ago:
Thanks, that's helpful real-world feedback (not that I wouldn't
also be interested in some synthetic benchmark comparisons from
someone else).
aggieNick02 wrote 2 hours 52 min ago:
We benchmarked three of the popular Optane NVMe SSDs about three
years ago. There was a short window when they were on clearance
and a popular choice as a cache SSD in TrueNAS. [1] You can
compare their benchmarks with the other almost 400 SSDs we've
benchmarked. Most impressive is that three years later they are
still the top random read QD1 performers, with no traditional
flash SSD coming anywhere close: [2] They are amazing for how
consistent and boring their performance is. Bit level access
means no need for TRIM or garbage collection, performance doesn't
degrade over time, latency is great, and random IO is not
problematic.
URI [1]: https://pcpartpicker.com/forums/topic/425127-benchmarkin...
URI [2]: https://pcpartpicker.com/products/internal-hard-drive/be...
speedgoose wrote 4 hours 16 min ago:
I configured a hetzner ax101 bare metal server with a 480GB 3d xpoint
ssd some years ago. Itâs used as the boot volume and it seems fast
despite the server being heavily over provisioned, but I canât
really compare because I donât have a baseline without.
rkagerer wrote 4 hours 17 min ago:
Before people claim it doesn't matter due to OS write buffering, I
should point out a) today's bloated software and the many-layered,
abstracted I/O stack it's built on tends to issue lots of unnecessary
flushes, b) read latency is just as important as write (if not
moreso) to how responsive your OS feels, particularly if the whole
thing doesn't fit in (or preload to) memory.
gigatexal wrote 4 hours 39 min ago:
Iâm still sad they discontinued them. Whatâs the alternative now
does anything come close?
walterbell wrote 3 hours 13 min ago:
Small sizes are on secondary market for ~$1/GB.
zozbot234 wrote 3 hours 1 min ago:
Which is a bargain compared to what DRAM costs today. If you just
include the bare minimum of DRAM for a successful boot and
immediately set up the entire "small" Optane drive as swap, that's
a viable workstation-class system for comparative peanuts. You
can't do this with NAND because the write workload of swap kills
the media (I suppose it becomes viable if you monitor SMART wearout
indicators and heavily overprovision the storage to leverage the
drive's pSLC mode, but you're still treating $~0.10/GB hardware as
a consumable and that will cost you) and of course you can't do it
with spinning rust because the media is too slow.
readitalready wrote 4 hours 40 min ago:
These are absolute beasts for database servers, and definitely needs to
make a comeback.
They suck for large sequential file access, but incredible for small
random access: databases.
gozzoo wrote 4 hours 40 min ago:
Maybe we can also mention the HP Memristor here.
jamiek88 wrote 3 hours 19 min ago:
Oh I was so excited for that. I devoured any news or blogs or rumours
about that immediately!
dangoodmanUT wrote 4 hours 44 min ago:
Optane was crazy good tech, it way just too expensive at the time for
mass adoption, but the benefits were so good.
Looking at those charts, besides the DWPD it feels like normal NVMe has
mostly caught up. I occassionally wonder where a gen 7/8(?) optane
would be today if it caught on, it'd probably be nuts.
exmadscientist wrote 4 hours 12 min ago:
The actual strength of Optane was on mixed workloads. It's hard to
write a flash cell (read-erase-write cycle, higher program voltage,
settling time, et cetera). Optane didn't have any of that baggage.
This showed up as amazing numbers on a 50%-read, 50%-write mix.
Which, guess what, a lot of real workloads have, but benchmarks don't
often cover well. This is why it's a great OS boot drive: there's so
much cruddy logging going on (writes) at the same time as reads to
actually load the OS. So Optane was king there.
lvl155 wrote 2 hours 41 min ago:
Itâs the best OS drive especially p5800x.
zozbot234 wrote 4 hours 38 min ago:
> besides the DWPD it feels like normal NVMe has mostly caught up.
So what you mean is that on the most important metric of them all for
many workloads, Flash-based NVMe has not caught up at all. When you
run a write heavy workload on storage with a limited DWPD (including
heavy swapping from RAM) higher performance actually hurts your
durability.
ece wrote 4 hours 50 min ago:
Fabs are expensive and all, but maybe running a right-sized fab could
have still been profitable at making optane for low-latency work that
it was so good at. Even moreso with RAM prices as they are.
downrightmike wrote 16 min ago:
PE popped already, now Private credit is already popping, having been
in too many bubbles, Datacenter bubble popped them. The whole system
just isn't talking about it because the media isn't doing their job
to report. There are no more good loans to make, the ROI is nil and
the interest rate is spot plus 5.5%, and businesses don't want to pay
10% interest on a loan. We're ina zombie cycle
FpUser wrote 5 hours 9 min ago:
I feel sorry about the situation. From my perspective Optane was a
godsend for databases. I was contemplating building a system. Could've
been a pinnacle of vertical scalability for cheap.
amelius wrote 5 hours 11 min ago:
For a good technical explanation at the physical level of a memory
cell:
URI [1]: https://pcper.com/2017/06/how-3d-xpoint-phase-change-memory-wo...
walterbell wrote 5 hours 12 min ago:
Related: "High-bandwidth flash progress and future" (15 comments), [1]
In an era of RAM shortages and quarterly price increases, Optane
remains viable for swap and CPU/GPU cache.
URI [1]: https://news.ycombinator.com/item?id=46700384
Weryj wrote 3 hours 4 min ago:
Iâve been considering buying 8x64g models and setting them as equal
priority swap disks (to mitigate the low throughput) for this exact
reason.
MrDrMcCoy wrote 1 hour 48 min ago:
Can confirm doing so is awesome. Get some slightly bigger ones and
partition them for additional use as zil. They're extremely
satisfying to use, and depressing to remember that we'll never see
their like again.
newsclues wrote 4 hours 55 min ago:
in an era of shortages, if there was an optane factory today ready to
print money...
walterbell wrote 4 hours 38 min ago:
Secondary market surplus pricing (~$1/GB) value accrues to the
buyer..
zozbot234 wrote 4 hours 29 min ago:
> (~$1/GB)
Isn't that actually crazy good, even insane value for the
performance and DWPD you get with Optane, especially with DRAM
being ~$15/GB or so? I don't think ~$1/GB NAND is anywhere that
good on durability, even if the raw performance is quite possibly
higher.
trollbridge wrote 4 hours 58 min ago:
Yeah, I've wondered if we might see a revival of this kind of
technology.
ashvardanian wrote 5 hours 13 min ago:
I don't have the inside scoop on Intel's current mess, but they
definitely have a habit of killing off their coolest projects.
brcmthrowaway wrote 1 hour 41 min ago:
Realsense too
hbogert wrote 5 hours 13 min ago:
It stands out, because it didn't sell. Which is weird because there
were some pretty big pros about using them. The latency for updating 1
byte was crazy good. Some databases or journals for something like zfs
really benefited from this.
mort96 wrote 1 hour 50 min ago:
I never understood what they're meant to do. Intel seemed to picture
some future where RAM is persistent; but they were never close to
fast enough to replace RAM, and the option to reboot in order to fix
some weird state your system has gotten itself into is a feature of
computers, not a problem to work around.
thesz wrote 2 hours 31 min ago:
In "databases and journals" you rarely update just one byte, you do a
transaction that updates data, several indexes and metadata. All of
that needs to be atomic.
Power failure can happen in between any of "1 byte updates with crazy
latencies." However small latency is, power failure is still faster.
Usually, there is a write ahead or some other log that alleviates the
problem, this log is usually written in streaming fashion.
What is good, though, is that "blast radius" [1] of failure is
smaller than usual - failed one byte write rarely corrupts more that
one byte or cache line. SQLite has to deal with 512 (and even more)
bytes long possible corruptions on most disks, with Optane it is not
necessarily so. So, less data to copy, scan, etc.
URI [1]: https://sqlite.org/psow.html
amluto wrote 4 hours 24 min ago:
Intel did a spectacularly poor job with the ecosystem around the
memory cells. They made two plays, and both were flops.
1. âOptaneâ in DIMM form factor. This targeted (I think) two
markets. First, use as slower but cheaper and higher density
volatile RAM. There was actual demand â various caching workloads,
for example, wanted hundreds of GB or even multiple TB in one server,
and Optane was a route to get there. But the machines and DIMMs never
really became available. Then there was the idea of using Optane
DIMMs as persistent storage. This was always tricky because the DDR
interface wasnât meant for this, and Intel also seems to have a lot
of legacy tech in the way (their caching system and memory
controller) and, for whatever reason, they seem to be barely capable
of improving their own technology. They had multiple serious false
starts in the space (a power-supply-early-warning scheme using NMI or
MCE to idle the system, a horrible platform-specific register to poke
to ask the memory controller to kindly flush itself, and the
stillborn PCOMMIT instruction).
2. Very nice NVMe devices. I think this was more of a failure of
marketing. If they had marketed a line of SSDs that, coupled with an
appropriate filesystem, could give 99% fsync latency of 5
microseconds and they had marketed this, I bet people would have
paid. But they did nothing of the sort â instead they just threw
around the term âOptaneâ inconsistently.
These days one could build a PCM-backed CXL-connected memory mapped
drive, and the performance might be awesome. Heck, I bet it
wouldnât be too hard to get a GPU to stream weights directly off
such a device at NVLink-like speeds. Maybe Intel should try it.
orion138 wrote 4 hours 13 min ago:
One of the many problems was trying to limit the use of Optane to
Intel devices. They should have manufactured and sold Optane memory
and let other players build on top of it at a low level.
amluto wrote 3 hours 50 min ago:
> Optane memory
Which âOptane memoryâ? The NVMe product always worked on
non-Intel. The NVDIMM products that I played with only ever
worked on a very small set of rather specialized Intel platforms.
I bet AMD could have supported them about as easily as Intel, and
Intel barely ever managed to support them.
wtallis wrote 3 hours 39 min ago:
The consumer "Optane memory" products were a combination of
NVMe and Intel's proprietary caching software, the latter of
which was locked to Intel's platforms. They also did two
generations of hybrid Optane+QLC drives that only worked on
certain Intel platforms, because they ran a PCIe x2+x2 pair of
links over a slot normally used for a single X2 or x4 link.
Yes, the pure-Optane consumer "Optane memory" products were at
a hardware level just small, fast NVMe drives that could be use
anywhere, but they were never marketed that way.
myself248 wrote 3 hours 21 min ago:
Exactly. I happen to have all AMD sitting around here, and
buying my first Optane devices was a gamble, because I had no
idea if they'd work. Only reason I ever did, is they got
cheap at one point and I could afford the gamble.
That uncertainty couldn't have done the market any favors.
amluto wrote 3 hours 28 min ago:
I feel like this is proving my point. You canât read
âOptaneâ and have any real idea of what youâre buying.
Also⦠were those weird hybrid SSDs even implemented by
actual hardware, or were they part of the giant series of
massive kludges in the âRapid Storageâ family where some
secret sauce in the PCIe host lied to the OS about what was
actually connected so an Intel driver could replace the
OSâs native storage driver (NVMe, AHCI, or perhaps
something worse depending on generation) to implement all the
actual logic in software?
It didnât help Intel that some major storage companies
started selling very, very nice flash SSDs in the mean time.
wtallis wrote 3 hours 20 min ago:
> were those weird hybrid SSDs even implemented by actual
hardware, or were they part of the giant series of massive
kludges
They were definitely part of the series of massive kludges.
But aside from the Intel platforms they were marketed for,
I never found a PCIe host that could see both of the NVMe
devices on the drive. Some hosts would bring up the x2 link
to the Optane half of the drive, some hosts would bring up
the x2 link to the QLC half of the drive, but I couldn't
find any way to get both links active even when the drive
was connected downstream of a PCIe switch that definitely
had hardware support for bifurcation down to x2 links. I
suspect that with appropriate firmware hacking on the host
side, it may have been possible to get those drives fully
operational on a non-Intel host.
cogman10 wrote 4 hours 27 min ago:
IMO, the reason they didn't sell is the ideal usage for them is
pairing them with some slow spinning disks. The issue Optane had is
that SSD capacity grew dramatically while the price plummeted. The
difference between Optane and SSDs was too small. Especially since
the M.2 standard proliferated and SSDs took advantage of PCI-E
performance.
I believe Optane retained a performance advantage (and I think even
today it's still faster than the best SSDs) but SSDs remain good
enough and fast enough while being a lot cheaper.
The ideal usage of optane was as a ZIL in ZFS.
exmadscientist wrote 4 hours 14 min ago:
> The ideal usage of optane was as a ZIL in ZFS.
It was also the best boot drive money could buy. Still is, I think,
though other comments in the thread ask how it compares against
today's best, which I'd also love to see.
gozzoo wrote 3 hours 49 min ago:
This concept was very popular back in the days when computers
used to boot from HDD, but now it doesn't make much sense. I
wouldn't notice If my laptop boots for 5 sec instead of 10.
exmadscientist wrote 3 hours 19 min ago:
At the time of their introduction Optane drives were noticeably
faster to boot your machine than even the fastest available
Flash SSD. So in a workstation with multiple hard drives
installed anyway, buying one to boot off of made decent sense.
If they had been cheaper, I think they'd have been really,
really popular.
bushbaba wrote 4 hours 21 min ago:
Not just capacity but SSD speeds also improved to the point it was
good enough for many high memory workloads.
zozbot234 wrote 4 hours 22 min ago:
That may have been the ideal usage back in the day, but ideal usage
now is just for setting up swap. Write-heavy workloads are king
with Optane, and threshing to swap is the prototypical example of
something that's so write-heavy it's a terrible fit for NAND.
Optane might not have been "as fast as DRAM" but it was plenty
close enough to be fit for purpose.
mort96 wrote 1 hour 47 min ago:
That would be fine if I could put it in an M.2 slot. But all my
computers already have RAM in their RAM slots, and even if I had
a spare RAM slot, I don't know that I'd trust the software stack
to treat one RAM slot as a drive...
And their whole deal was making RAM persistent anyway, which
isn't exactly what I want.
zozbot234 wrote 1 hour 43 min ago:
Optane M.2-format hardware exists.
mort96 wrote 1 hour 20 min ago:
Interesting, all I ever saw advertised was that weird
persistent kinda slow RAM stick. Does the M.2 version just
show up as a normal block device or is that too trying to be
persistent RAM?
ksec wrote 4 hours 51 min ago:
>Which is weird....
It isn't weird at all. I would be surprised if it ever succeed in the
first place.
Cost was way too high. Intel not sharing the tech with others other
than Micron. Micron wasn't committed to it either, and since unused
capacity at the Fab was paid by Intel regardless they dont care. No
long term solution or strategy to bring cost down. Neither Intel or
Micron have a vision on this. No one wanted another Intel only tech
lock in. And despite the high price, it barely made any profits per
unit compared to NAND and DRAM which was at the time making historic
high profits. Once the NAND and DRAM cycle went down again cost /
performance on Optane wasn't as attractive. Samsung even made some
form of SLC NAND that performs similar to Optane but cheaper, and
even they end up stopped developing for it due to lack of interest.
amluto wrote 1 hour 23 min ago:
A ways back, I wrote a sort of database that was memory-mapped-file
backed (a mistake, but I didnât know that at the time), and I
would have paid top dollar for even a few GB of NVDIMMs that could
be put in an ordinary server and could be somewhat
straightforwardly mounted as a DAX filesystem. I even tried to do
some of the kernel work. But the hardware and firmware was such a
mess that it was basically a lost cause. And none of the tech ever
seemed to turn into an actual purchasable product. Iâm a bit
suspicious that Intel never found product-market fit in part
because they never had a credible product on the NVDIMM side.
Somewhere I still have some actual battery-backed DIMMs (DRAM plus
FPGA interposer plus awkward little supercapacitor bundle) in a
drawer. They were not made by Intel, but Intel was clearly using
them as a stepping stone toward the broader NVDIMM ecosystem. They
worked on exactly one SuperMicro board, kind of, and not at all if
you booted using UEFI. Rebooting without doing the magic handshake
over SMBUS [0] first took something like 15 minutes, which was not
good for those nines of availability.
[0] You can find my SMBUS host driver for exactly this purpose on
the LKML archives. It was never merged, in part, because no one
could ever get all the teams involved in the Xeon memory controller
to reach any sort of agreement as to who owned the bus or how the
OS was supposed to communicate without, say, defeating platform
thermal management or causing the refresh interval to get out of
sync with the DIMM temperature, thus causing corruption.
Iâm suspicious that everything involved in Optane development was
like this.
deepsquirrelnet wrote 4 hours 2 min ago:
I worked at Micron in the SSD division when Optane (originally
called crosspoint âXpointâ) was being made. In my mind, there
was never a real serious push to productize it. But itâs not
clear to me whether that was due to unattractive terms of the joint
venture or lack of clear product fit.
There was certainly a time when it seemed they were shopping for
engineers opinions of what to do with it, but I think they quickly
determined it would be a much smaller market anyway from ssds and
didnât end up pushing on it too hard. I could be wrong though,
itâs a big company and my corner was manufacturing and not
product development.
rjsw wrote 1 hour 31 min ago:
A friend was working at Micron on a rackmount network server with
a lot of flash memory, I didn't ask at the time what kind of
flash it used. The project was cancelled when nearly finished.
chrneu wrote 3 hours 31 min ago:
I worked at Intel for a while and might be able to explain this.
There were/are often projects that come down from management that
nobody thinks are worth pursuing. When i say nobody, it might not
just be engineers but even say 1 or 2 people in management who
just do a shit roll out. There are a lot of layers of Intel and
if even one layer in the Intel Sandwich drag their feet it can
kill an entire project. I saw it happen a few times in my time
there. That one specific node that intel dropped the ball on kind
of came back to 2-3 people in one specific department, as an
example.
Optane was a minute before I got there, but having been excited
about it at the time and somewhat following it, that's the vibe I
get from Optane. It had a lot of potential but someone screwed it
up and it killed the momentum.
empiricus wrote 2 hours 58 min ago:
This is actually insane. Do you mean 2-4 people in one
department basically killed Intel? Roll to disbelief.
LASR wrote 1 hour 33 min ago:
Yes this is pretty common in large enterprise-ey tech
companies that are successful. There are usually a small
group of vocal members that have a strong conviction and
drive to make a vision a reality. This is contrary to popular
belief that large companies design by committee.
Of course it works exceptionally well when the instinct turns
out to be right. But can end companies if it isnât.
wtallis wrote 2 hours 44 min ago:
It's somewhat plausible that a small group of people in one
department were responsible for the bad bets that made their
10nm process a failure. But it was very much a group effort
for Intel to escalate that problem into the prolonged
disaster. Management should have stopped believing the
undeliverable promises coming out of their fab side after a
year or two, and should have started much sooner to design
chips targeting fab processes that actually worked.
osnium123 wrote 3 hours 18 min ago:
Are you referring to the Intel 10nm struggles in your reference
to 2-3 people?
jauntywundrkind wrote 4 hours 31 min ago:
Cost was fantastically cheap, if you take into account that Optane
is going to live >>10x longer than a SSD.
For a lot of bulk storage, yes, you don't have frequently changing
data. But for databases or caches, that are under heavy load,
optane was not only far faster, but if looking at life-cycle costs,
way way less.
mapt wrote 1 hour 9 min ago:
Write endurance of the drive would be measured in TBW, and TLC
flash kept adding enough 3D layers to stay cheap enough, quickly
enough, that Optane never really beat their pricing per TBW to
make a practical product.
I have to wonder if it isn't usable for some kind of specialized
AI workflow that would benefit from extremely low latency reads
but which is isn't written often, at this point. Perhaps
integrated in a GPU board.
zozbot234 wrote 32 min ago:
Optane practical TBW endurance is way higher than that of even
TLC flash, never mind QLC or PLC which is the current standard
for consumer NAND hardware. It even seems to go way beyond
what's stated on the spec sheet. However, while Optane excels
for write-heavy workloads (not read-heavy, where NAND actually
performs very well) these are also power-hungry which is a
limitation for modern AI workflow.
jauntywundrkind wrote 33 min ago:
The extra capacity of modern SSD is a good point, especially
now that we have 100TB+ SSD.
But Optane still offered 100 DWPD (drive writes per day), up to
3.2TB. Thats still just so many more DWPD than flash ssd. A
Kioxia CM8V for example will do 12TB at 3 DWPD. The net TBW is
still 10x apart.
You can get back to high endurance with SLC drives like the
Solidigm p7-p5810, but you're back down to 1.6TB and 50 DWPD,
so, 1/4 the Intel P5800X endurance, and worse latencies. I
highly suspect the drive model here is a homage, and in spite
of being much newer and very expensive, the original is still
so much better in so many ways. [1] You also end up paying for
what I assume is a circa six figure drive, if you are
substituting DWPD with more capacity than you need. There's
something elegant about being able to keep using your cells,
versus overbuying on cells with the intent to be able to rip
through them relatively quickly.
URI [1]: https://www.solidigm.com/content/solidigm/us/en/produc...
wtallis wrote 3 hours 44 min ago:
Optane was in the market during a time when the mainstream trend
in the SSD industry was all about sacrificing endurance to get
higher capacity. It's been several years, and I'm not seeing a
lot of regrets from folks who moved to TLC and QLC NAND, and
those products are more popular than ever.
The niche that could actually make use of Optane's endurance was
small and shrinking, and Intel had no roadmap to significantly
improve Optane's $/GB which was unquestionably the technology's
biggest weakness.
zozbot234 wrote 4 hours 55 min ago:
Optane didn't sell because they focused on their weird persistent
DIMM sticks, which are a nightmare for enterprise where for many
ordinary purposes you want ephemeral data that disappears as soon as
you cut power. Thet should have focused on making ordinary storage
and solving the interconnect bandwidth and latency problems
differently, such as with more up-to-date PCIe standards.
hrmtst93837 wrote 2 hours 7 min ago:
PCIe was a bottleneck in consumer boxes, but that wasn't the whole
problem. Optane's low latency and write endurance looked great on
paper, yet once you put it behind SSD controllers and file systems
built around NAND assumptions, a lot of the upside got shaved off
before users ever saw it.
"Just make it a faster SSD" was never a business. The DIMMs were
weird, sure, but the bigger issue was that Optane made the most
sense when software treated storage and memory as one tier, and
almost nobody was going to rewrite kernels, DBs, and apps for a
product that cost more than flash and solved pain most buyers
barely felt.
jauntywundrkind wrote 4 hours 35 min ago:
I don't think that would be my main complaint. Sticking optane in a
dimm was just awkward as hell. You now have different bits of
memory with very different characteristics, & you lose a ton of
bandwidth.
If CXL was around at the time it would have been such a nice fit,
allowing for much lower latency access.
It also seems like in spite of the bad fit, there were enough
regular options drives, and they were indeed pretty incredible.
Good endurance, reasonable price (and cheap as dirt if you consider
that endurance/lifecycle cost!), some just fantastic performance
figures. My conclusion is that alas there just aren't many people
in the world who are serious about storage performance.
tayo42 wrote 3 hours 54 min ago:
Can Linux differentiate that different dimms are different? Or
does it see it all as one big memory space still?
wmf wrote 1 hour 27 min ago:
Yes, Linux was aware of the difference via ACPI tables.
p-e-w wrote 5 hours 3 min ago:
Optane was a victim of its own hype, such as âentirely new
physicsâ, or âas fast as RAM, but persistentâ. The reality felt
like a failure afterwards even though it was still revolutionary,
objectively speaking.
epistasis wrote 5 hours 3 min ago:
When most people are running databases on AWS RDS, or on ridiculous
EBS drives with insanely low throughput and latency, it makes sense
to me.
There are very few applications that benefit from such low latency,
and if one has to go off the standard path of easy, but slow and
expensive and automatically backup up, people will pick the ease.
Having the best technology performance is not enough to have product
market fit. The execution required from the side of executives at
Intel is far far beyond their capability. They developed a platform
and wanted others to do the work of building all the applications.
Without that starting killer app, there's not enough adoption to
build an ecosystem.
amluto wrote 3 hours 25 min ago:
> There are very few applications that benefit from such low
latency
Basically any RDBMS? MySQL and Postgres both benefit from high
performance storage, but too many customers have moved into the
cloud where you canât get NVMe-like performance for durable
storage for anything remotely close to a worthwhile price.
epistasis wrote 3 hours 7 min ago:
I'm saying that there are very few downstream applications that
use databases that benefit from reducing latency beyond the slow
performance of the cloud. Running your database on VMs or
baremetal gives better performance, but almost no applications
built on databases bother to do it.
bombcar wrote 5 hours 4 min ago:
It feels like everyone figured out what to do with them and how just
about when they stopped making them.
timschmidt wrote 4 hours 56 min ago:
Same for the Larabee / Knights architecture. Would sure be fun to
play around with a 500 core Knights CPU with a couple TB of optane
for LLM inference.
Intel's got an amazing record of axing projects as soon as they've
done the hard work of building an ecosystem.
zozbot234 wrote 4 hours 51 min ago:
> 500 core
The newest fully E-core based Xeon CPUs have reached that figure
by now, at least in dual-socket configs.
timschmidt wrote 4 hours 45 min ago:
Yup. And high end GPU compute now has on-package HBM like
Knight's had a decade ago, and those new Intel CPUs are finally
shipping with AVX reliably again. We lost a decade for
workloads that would benefit from both.
DIR <- back to front page