![]() ![]() I was running into issues with running out of disk space with a 30 GB zipped tar file. While not the most elegant, especially since it's not copying a single zip or tar file and doubly so as it doesn't help to reduce network ovehead, my only choice was to use scp -r:įollows symbolic links encountered in the tree traversal. This is often unnecessary (because decompression and hard disk writing speed is most time higher than the network speed), but does usually not harm, either. In the example a buffer is also used on the receiver side. Pv is then used again, but for buffering ( q for quiet, C for the no splice mode and B to set the buffer size). xz is slow but compresses very well (good over slow connections), pigz/gzip is fast but compresses not so well. ![]() Instead of zstd, one can also use xz or pigz. If not available, it can be compiled and installed from with just make -j4 & sudo make install. This version is necessary for proper multi core and long support. One should be aware that together with the long mode these settings use a lot of memory.Ī problem with zstd is that most operating systems don't ship with version >= 1.3.4. The T0 options sets the amount of threads to the number of cores. In the example, the long mode 31 (uses a 2 GB window, needs a lot of RAM, but very good e.g. With four cores on a Haswell 3.2 GHz CPU 14 gives a speed of around 120 MB/s. This number can be reduced or increased depending on the network and CPU speed so zstd is a little bit faster than the the network speed. Zstd is used with the compression setting 14. The progress is measured before compression and buffering, therefore it's not very accurate, but still helpful. The total size is estimated with du and added to the size option ( s). The first pv is to show the progress ( p), estimated time ( e), transfer rate ( r), average rate ( a), total transferred bytes ( b). Here is an example to send data with a progress bar, buffering and zstandard compression over a network: tar cf. To use gzip with multiple cores pigz can be used. New xz and zstd have multi core support already built in. Gzip (like used above) is a fast compression algorithm, but in general zstandard ( zstd) (and for high compression ratios LZMA/LZMA2 ( xz) will compress better and be faster the same time. This can be done with with pv.Īdditionally, one can usually increase the speed with a proper compression algorithm. Therefore, it helps to buffer the data before sending them over the network. Often the network is the bottleneck and the speed varies over time. Answer is good, but one can increase the speed with a buffer and good compression and add a progress bar. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |