Sami Elkady

Category: General

  • File compression in Linux

    File compression in Linux

    For along while I was researching the history of compression tools for Linux and I would like to share some of it. It is helpful to know the commands used to manipulate the different formats. This article will at least show the basic/general method of performing a compression and extraction, plus various other tricks.

    some of these is new to me let’s go

    tar – The tar archive is the most popular archive format for Unix and Unix-like systems. A tarball is a tar file. Most tarballs are compressed with some other compression format (the new file is still called a tarball). The reason for this is most compression formats are unable to compress multiple files together as one file. Also, Tar cannot compress files, so using Tar with a compression format solves such issues. Tarballs have various file extensions as seen below.

    Short (Long) – Compression

    .taz (.tar.z) – zip
    .tbz, .tb2, .tbz2 (.tar.bz2) – bzip2
    .tgz (.tar.gz) – gzip
    .tlz (.tar.lzma, .tar.lz) – lzma
    .txz (.tar.xz) – xz

    Archive:

    Code:
    tar -cf NEW_ARCHIVE.tar FILE

    Extract:

    Code:
    tar -xf ARCHIVE.tar #regular tar
tar -xfz ARCHIVE.tar.gz #compressed -> uncompress -> untar

    List Contents:

    Code:
    tar -tvf ARCHIVE.tar

    Compression Files:
    Files that compress data so the resulting file consumes less storage (bytes).

    bz2 – Bzip2 offers more efficient compression than zip or gzip. Bzip2 can only compress one file. That means multiple files compresses with bzip2 will produce individual bzip2 files. However, Tar is used to archive the files into one (the *.tar file) and Bzip2 compresses this file. That is why a tar extension is on most bzip2 files. The “bzcat” command can be used on Bzip2 files the same way “cat” is used on regular files.

    Compress:

    Code:
    bzip2 -z NEW_ARCHIVE.bz2 FILE

    Extract:

    Code:
    bzip2 -d FILE.bz2 #bunzip2 = "bzip2 -d"

    gz – GNU Zip (Gzip) is a popular compression format that is faster but less efficient than Bzip2. Like Bzip2, Gzip can only compress one file. This is why Tar is often used with Gzip. Gzip was made to replace the “compress” utility/command in earlier Unix systems. To view the contents of files inside of Gzip files like using “cat” on other files, use the “gzcat” command. Gzip is the most popular compression utility in Gnu/Linux, so it is highly recommended that user familiarize themselves with Gzip’s commands.

    Compress:

    Code:
    gzip -z NEW_ARCHIVE.gz FILE

    Extract:

    Code:
    gunzip FILE.gz

    zip – ZIP is a widely used compression format. ZIP supports password protected archives.

    Compress:

    Code:
    zip ARCHIVE_NAME FILES

    Extract:

    Code:
    unzip ARCHIVE_NAME

    rar – The Roshal Archive (RAR) is a commonly seen format, especially among Windows users. RAR is not open-source, but it is free. RAR supports password protection.

    Compress:

    Code:
    rar a NEW_ARCHIVE.rar FILES

    Extract:

    Code:
    rar x FILE.rar

    ISO – An ISO file is a optical disc image or a file that mimics an optical-disc (think of virtual machines).

    Archive:

    Code:
    dd if=/dev/cdrom of=/tmp/cdimg1.iso

    “if=/dev/cdrom” reads the optical disc and “of=/tmp/cdimg1.iso” is where the data is written. The ISO file will contain the disc’s data including the filesystem.

    Extract:

    Code:
    mkdir /mnt/iso #make a directory
mount -o loop FILE.iso /mnt/iso #Mount your iso file (FILE.iso) to the new directory
cp * ~/ISO/ #copy all files in the ISO to a folder in $HOME

    7z – The 7z compression format was first used in an application named “7-Zip”, hence the filetype’s name. 7z can compress multiple files together without the need of tar. 7z does not store file permissions, unlike other compression formats.

    Compress:

    Code:
    7za a NEW_ARCHIVE.7z FILES

    Extract:

    Code:
    7za e FILE.7z

    ar – ARchiver (ar) is the predecessor to tar. Debian packages (*.deb) still use ar.

    Archive:

    Code:

    ar rcs NAME_ARCHIVE_FILE FILES

    Extract:

    Code:
    ar -x ARCHIVE.a

    CPIO – “CoPy In and Out” was once used on magnetic tape drives. CPIO is used today in for simple backups (other uses exist). The tar utility has largely replaced CPIO. The creation of CPIO files is more complicated than other archive formats. By that, I mean, the command is lengthier than other commands. As you read on, you will see that using CPIO is not like other commands.

    This may be a little off topic, but RPM files can be converted to CPIO files using the “rpm2cpio” command like this – “rpm2cpio FILE.rpm”. Not all systems have rpm2cpio.

    Archive:

    Code:
    FILE-LIST | cpio -o > ./FILE.cpio

    “FILE-LIST” may be the echo command with the file paths or a “find” command that will output a list of files.
    Extract:

    Code:
    cpio -id < FILE.cpio

    List Contained Files:

    Code:
    cpio -it < FILE.cpio

    Shell Archive (shar) – A shar file is a self-extracting archive that only depends on the sh utility. This file is a plain-text shell script that recreates the files on execution. This format is not secure. If the user desires an archive format to hide the contained files, do not use shar files. Obviously, since this is self-extracting, the file needs to be executed like any other application/script.

    Archive:

    Code:
    shar FILES

    Freeze (F) – Freeze is a compression format that can only compress one file (just like Bzip2). To view the contents of a file (like a text file) compressed as Freeze, use the “fcat” command – “fcat FILE.f | less”

    Compress:

    Code:
    freeze -x FILE #the new file will be FILE.f

    NOTE: The “-x” makes the compression better at the cost of speed. This parameter can be removed if desired.

    Extract:

    Code:
    freeze -d FILE.F #melt = unfreeze = "freeze -d"

    lzma – The Lempel–Ziv–Markov chain algorithm (LZMA) compresses data better than Bzip2 and Gzip, but is slower than those two algorithms. Like Bzip2 and Gzip, LZMA can only compress one file which is why Tars are used when compressing many files (the *.tar counts as one file). The command “lzcat” acts like “cat” by showing users the contents of the compressed files.

    Compress:

    Code:
    xz --format=lzma NEW_ARCHIVE.lzma FILE

    Extract:

    Code:
    unlzma FILE.lzma

    TIP: Make an alias for lzma to make it easier to use – “alias lzma=’xz –format=lzma ‘”. With this alias, users can type “lzma” instead of “xz –format=lzma”.

    NOTE: Using the different compression cat-like commands (such as gzcat and others) even works on tar-containing files (like *.tar.lzma).

    xz – XZ uses the LZMA2 algorithm thus making it better than LZMA. Again, like other compression formats, to compress multiple files, use Tar and then XZ. The “xzcat” command is a cat-like command that allows users to view the contents of the compressed files.

    Compress:

    Code:
    xz NEW_ARCHIVE.xz FILE

    Extract:

    Code:
    unxz FILE.xz

    Compression + Storage Files:
    Files that store and compress files. These formats do not require tar when compressing multiple files into one file.

    apk – Android installation packages are apk files. Android uses *.apk files just as Debian uses *.deb and Fedora uses *.rpm. APK files are created in the Android developer’s IDE.

    arc – Arc does not require multiple file be merged together (tar) to be compressed.

    Compress:

    Code:
    arc NEW_ARCHIVE.arc FILES

    Extract:

    Code:
    arc x FILE.arc

    jar – Java classes (compiled Java-code files) are held together using a Jar file. Jar is not intended to be used as storage. It is meant to be an application file containing executable code. Jar files can also be used as addons/plugins for various applications.

    Compress:

    Code:
    jar cf NEW_ARCHIVE.jar FILES

    Extract:

    Code:
    jar xf FILE.jar

    kgb – This compression archive uses the PAQ6 algorithm to gain high compression ratios. Self-extracting archives and AES-256 encryption is supported.

    Compress:

    Code:
    kgb ARCHIVE.kgb FILES

    xar – eXtensible ARchive format is open-source archiving format. RPM5 uses xar.

    Compress:

    Code:
    xar -cf FILE.xar FILES

    Extract:

    Code:
    xar -xf FILE.xar

    zpaq – ZPAQ is a special type of compressor. When a newer version of a file is added to a ZPAQ file that contains the older version, ZPAQ performs an incremental update. This means the older version of the file exists in the archive.

    Compress:

    Code:
    zpaq ARCHIVE_NAME FILES

    zz – Zzip is a self-extracting archive. This means the file is “executed” and it will uncompress itself.

    Compress:

    Code:
    zzip ARCHIVE_NAME FILES

    Usage: gzip [OPTION]… [FILE]…
    Compress or uncompress FILEs (by default, compress FILES in-place).

    Mandatory arguments to long options are mandatory for short options too.

    -c, –stdout write on standard output, keep original files unchanged
    -d, –decompress decompress
    -f, –force force overwrite of output file and compress links
    -h, –help give this help
    -k, –keep keep (don’t delete) input files
    -l, –list list compressed file contents
    -L, –license display software license
    -n, –no-name do not save or restore the original name and time stamp
    -N, –name save or restore the original name and time stamp
    -q, –quiet suppress all warnings
    -r, –recursive operate recursively on directories
    -S, –suffix=SUF use suffix SUF on compressed files
    -t, –test test compressed file integrity
    -v, –verbose verbose mode
    -V, –version display version number
    -1, –fast compress faster
    -9, –best compress better
    –rsyncable Make rsync-friendly archive

    With no FILE, or when FILE is -, read standard input.

    Report bugs to <bug-gzip@gnu.org>.

    Compression Tool File Extension Decompression Tool
    bzip2 .bz2 bunzip2
    gzip .gz gunzip
    zip .zip unzip
    tar

  • USER VISUDO

    Use the NOPASSWD directive

    You can use the NOPASSWD directive in your /etc/sudoers file.

    If your user is called user and your host is called host you could add these lines to /etc/sudoers:

    user host = (root) NOPASSWD: /sbin/shutdown
user host = (root) NOPASSWD: /sbin/reboot

    This will allow the user user to run the desired commands on host without entering a password. All other sudoed commands will still require a password.

    The commands specified in the sudoers file must be fully qualified (i.e. using the absolute path to the command to run) as described in the sudoers man page. Providing a relative path is considered a syntax error.

    If the command ends with a trailing / character and points to a directory, the user will be able to run any command in that directory (but not in any sub-directories therein). In the following example, the user user can run any command in the directory /home/someuser/bin/:

    user host = (root) NOPASSWD: /home/someuser/bin/

    Note: Always use the command visudo to edit the sudoers file to make sure you do not lock yourself out of the system – just in case you accidentally write something incorrect to the sudoersfile. visudo will save your modified file to a temporary location and will only overwrite the real sudoers file if the modified file can be parsed without errors.

    Using /etc/sudoers.d instead of modifying /etc/sudoers

    As an alternative to editing the /etc/sudoers file, you could add the two lines to a new file in /etc/sudoers.d e.g. /etc/sudoers.d/shutdown. This is an elegant way of separating different changes to the sudo rights and also leaves the original sudoers file untouched for easier upgrades.

    Note: Again, you should use the command visudo to edit the file to make sure you do not lock yourself out of the system:

    sudo visudo -f /etc/sudoers.d/shutdown

    This also automatically ensures that the owner and permissions of the new file is set correctly.

    If sudoers is messed up

    If you did not use visudo to edit your files and then accidentally messed up /etc/sudoers or messed up a file in /etc/sudoers.d then you will be locked out of sudo.

    The solution could be to fix the files using pkexec which is an alternative to sudo.

    To fix /etc/sudoers:

    pkexec visudo

    To fix /etc/sudoers.d/shutdown:

    pkexec visudo -f /etc/sudoers.d/shutdown

    If the ownership and/or permissions are incorrect for any sudoers file, the file will be ignored by sudo so you might also find yourself locked out in this situation. Again, you can use pkexec to fix this.

    The correct permissions should be like this:

    $ ls -l /etc/sudoers.d/shutdown 
-r--r----- 1 root root 86 Jul 16 15:37 /etc/sudoers.d/shutdown

    Use pkexec like this to fix ownership and permissions:

    pkexec chown root:root /etc/sudoers.d/shutdown
pkexec chmod 0440 /etc/sudoers.d/shutdown

    https://www.digitalocean.com/community/tutorials/how-to-edit-the-sudoers-file-on-ubuntu-and-centos