_______               __                   _______
       |   |   |.---.-..----.|  |--..-----..----. |    |  |.-----..--.--.--..-----.
       |       ||  _  ||  __||    < |  -__||   _| |       ||  -__||  |  |  ||__ --|
       |___|___||___._||____||__|__||_____||__|   |__|____||_____||________||_____|
                                                             on Gopher (inofficial)
   URI Visit Hacker News on the Web
       
       
       COMMENT PAGE FOR:
   URI   Lizard – efficient compression with fast decompression
       
       
        p0nce wrote 3 days ago:
        Would be interesting to couple it with QOI, as QOI + LZ4 is very
        interesting as image format.
       
          genewitch wrote 2 days ago:
          as in you run the bitmap through QOI and then compress that output
          with LZ4? Or am i misremembering how QOI works and it uses LZ4
          internally? PNG uses LZ, doesn't it?
       
            adgjlsfhk1 wrote 2 days ago:
            QOI doesn't use any entropy compression.
       
            p0nce wrote 2 days ago:
            > as in you run the bitmap through QOI and then compress that
            output with LZ4?
            
            Yes, it can be very beneficial.
       
        lifthrasiir wrote 4 days ago:
        Note that a benchmark in the README refers to zstd 1.1.1 and brotli
        0.5.2, which are very old (the current versions are zstd 1.5.2 and
        brotli 1.0.9). The same author maintains lzbench [1] which is more
        current than the README. It shows that Lizard heavily trades off
        compression speed and ratio for decompression speed; zstd at the
        default setting (-3) is expected to be on par with Lizard -45 in terms
        of ratio, but Lizard would be >10x slower to compress while only ~3x
        faster to decompress.
        
   URI  [1]: https://github.com/inikep/lzbench
       
        deepsun wrote 4 days ago:
        My take (I compare about ZSTD only, as that what we need to care
        about):
        1. Lizard's decompression speeds are much higher than ZSTD for any
        level of compression.
        2. Lizards's compression speed is significantly (several times) lower
        than ZSTD for similar compression ratios.
        
        So it makes sense if we mostly don't care about CPU usage for
        compression.
       
          CoolGuySteve wrote 4 days ago:
          The thing I don't get about zstd is that their own github page shows
          lz4 is faster at both compression and decompression at the cost of
          some compression ratio: [1] But most people I work with will pick
          zstd every time even in cases where decompression speed matters the
          most.
          
   URI    [1]: https://github.com/facebook/zstd
       
            loxias wrote 4 days ago:
            > But most people I work with will pick zstd every time even in
            cases where decompression speed matters the most.
            
            Zstd is really good.  If you're a math/compression theory nerd it
            makes a lot of good choices.  Also, if you're willing to tune zstd
            instead of using the defaults, so far anyway, I haven't found any
            application where the right answer wasn't zstd.
            
            At this point I don't think it's unreasonable to expect new
            compression developments to to explicitly say "What is improved
            over ZStandard".
       
              CoolGuySteve wrote 3 days ago:
              That's the thing, even at its fastest setting, zstd is still
              slower than lz4.  So clearly, it's not the right answer if you're
              serializing from a fast medium.
              
              And this is exactly the kind of cognitive laziness I'm talking
              about.
       
                adgjlsfhk1 wrote 3 days ago:
                Note that ZSTD decompression is generally faster for data
                compressed with slower settings.
       
                  deepsun wrote 3 days ago:
                  Nope, as I wrote above, ZSTD decompression is always slower
                  than Lizard's, regardless of settings.
       
                    adgjlsfhk1 wrote 2 days ago:
                    sorry for the confusion. I meant that zstd at higher
                    compression decompresses faster than zstd at lower
                    compression.
       
                arinlen wrote 3 days ago:
                > That's the thing, even at its fastest setting, zstd is still
                slower than lz4. So clearly, it's not the right answer if
                you're serializing from a fast medium.
                
                It really depends on what you want to do, as decompression
                speeds can be meaningless and irrelevant in some applications,
                if not practically all of them.
                
                Can you provide a specific example where you feel lz4 is hands
                down the best option?
       
                  CoolGuySteve wrote 3 days ago:
                  Yes, I can and did.  When serializing off a fast medium like
                  nvme drives.
                  
                  And again, I shouldn’t have to keep explaining this.    zstd
                  isn’t some magic answer to everything.
       
                    Dylan16807 wrote 2 days ago:
                    > I shouldn’t have to keep explaining this. zstd isn’t
                    some magic answer to everything.
                    
                    That logic doesn't really work.  It's true that zstd
                    doesn't let you tune it to be as fast as lz4.  But it's
                    possible that a slightly different design could make that
                    possible, so that the new version is better than lz4 at all
                    speeds and at all ratios.  Making one compression algorithm
                    strictly better than another requires no magic.
       
              ciupicri wrote 3 days ago:
              Just to add to what you're saying, you can't tune zstd for qcow
              images or for btrfs filesystems.
       
                loxias wrote 3 days ago:
                You're entirely right!    And points out the blind spot that I've
                used it much more as a library in tools I build/use than
                integrated with other things.  Never used brtfs, I'm assuming
                it uses zstd internally for something?    Cool.
                
                Also very cool that qcow contains zstd support (didn't know
                that either!).
       
                  ciupicri wrote 2 days ago:
                  From [1] > Btrfs supports transparent file compression. There
                  are three algorithms available: ZLIB, LZO and ZSTD(since
                  v4.14)
                  
                  I've been using zstd compression for qcow2 images since last
                  year, but I admit it's kind of hidden. The man page for
                  qemu-img [1] doesn't say a word about zstd, but it's there
                  alright [2].
                  
                  [1]: [2]:
                  
   URI            [1]: https://btrfs.wiki.kernel.org/index.php/Compression
   URI            [2]: https://www.qemu.org/docs/master/tools/qemu-img.html
   URI            [3]: https://gitlab.com/qemu-project/qemu/-/blob/v7.0.0/b...
       
            adgjlsfhk1 wrote 4 days ago:
            If you are bottle-necked by reading data (eg over networks or slow
            storage), zstd will be faster to decompress since you are
            transferring less data.
       
              arinlen wrote 3 days ago:
              > If you are bottle-necked by reading data (eg over networks or
              slow storage), zstd will be faster to decompress since you are
              transferring less data.
              
              This should be stressed. Benchmarks show zstd's compression ratio
              being 75% of lz4's. Pushing 25% less data over a network is
              enough to win any real-world test, not to mention the impact it
              has on operational costs.
       
                woobilicious wrote 3 days ago:
                I have this pathological example, I have a 30GB journal dump
                from bcachefs, and compressed it down with lz4 and zstd to a
                sata SSD, the compression ratios are 6.8% and 3% respectively
                (and this with default level 3!), it takes 14s to ripgrep
                through the lz4 version, and 11s with the zstd version. So even
                at 500MB/s zstd is faster for decompression.
       
                  adgjlsfhk1 wrote 3 days ago:
                  LZ4 requires about 2gb/s to be competitive on any relatively
                  recent desktop/laptop cpu
       
          dataangel wrote 4 days ago:
          if it's faster for both compression and decompression it just seems
          better period? why do you say "if we mostly don't care about CPU
          usage"? being faster means less CPU usage...
       
            kordlessagain wrote 4 days ago:
            Faster/slower aside, it doesn't matter if it is slower to compress,
            because you only do that once for most use cases (thinking of
            downloadable repos or document stores). Decompression happens much
            more frequently. So, one compression algo that takes a lot of work
            (and time) to compress but uncompresses with less work than
            anything else is desirable because of the fact you only need
            compress once. So, less compute is required.
            
            There could be some use cases where this doesn't work out, such as
            a compressed network connection, where everything has to be
            compressed/decompressed in a 1:1 ratio.
       
            deepsun wrote 4 days ago:
            I said it's significantly _slower_, not faster, for compression.
       
              metadat wrote 4 days ago:
              Minor nit: I too was confused by the wording:
              
              "compression speed is lower"
              
              To me this reads as though it's faster.  Maybe you meant
              "compression speed is Slower"?
              
              Disclaimer: I generally dislike being pedantic, especially on HN.
               I hope this message doesn't come across that way.
       
                darrenf wrote 4 days ago:
                If compression speed is lower, then it's slower.
                
                If compression time is lower, then it's faster.
       
       
   DIR <- back to front page