2019 oplossingen labo 0 - Lars Lemmens

Met dank aan de Github van Martijn en natuurlijk Lars Lemmens

LABO 0

Exercise 24

Create 2 files with random text. Then create a third file containing the contents of the two other files

1) echo 'some random text' > file1
2) pwgen -n1 > file2 
3) cat file1 file2 > file3

• The pwgen command generates passwords which are designed to be easily memorized by humans, while being as secure as possible.
• -n stands for minimum 1 number AND -1 stands for per line 1 password.
• '>' means overwrite a file

Exercise 25

List all home directories which are open for other users (group or others should have r,w or x rights)

• ls -l /home/LDAP/ | grep -v 'drwx------' | grep -v '^total' | tr -s ' ' | cut -d ' ' -f9
• ls -l /home/LDAP/ | grep -v 'drwx------' | grep -v '^total' | awk '{print $9}'
• ls -l /home/LDAP/ | grep -vP '^(drwx------|total)'| awk '{print $9}'
• ls -l /home/LDAP/ | awk '$1 !~ /drwx------|total/{print $9}'

• grep prints lines matching a pattern.
• -v stands for inverted match.
• -P stands for perl expression
• ^ matches position just before the first character of the string.
• The tr command replaces all occurrences of a character in a file, and print the result.
• -s replaces each input sequence of a repeated character that is listed in SET1 with a single occurrence of that character
• The command cut removes sections from each line of files
• -d use DELIM instead of TAB for field delimiter
• -f(number) select only these fields
• awk '{print $(number)' prints the field with the matching number

Exercise 26

List all lines in the file /usr/share/dict/dutch containing the string 'rare'

• cat /usr/share/dict/dutch | grep rare
• grep rare /usr/share/dict/dutch

Exercise 27

Only show the columns with filenames and permissions of the files in your homedirectory. (tip: use ls and cut)

• ls -l ~ | grep -v '^total' | tr -s ' ' | cut -d ' ' -f1,9

• grep prints lines matching a pattern.
• -v stands for inverted match.
• The tr command replaces all occurrences of a character in a file, and print the result.
• -s replaces each input sequence of a repeated character that is listed in SET1 with a single occurrence of that character
• The command cut removes sections from each line of files
• -d use DELIM instead of TAB for field delimiter
• -f(number) select only these fields

Exercise 28

Sort the lines from 27 in reverse alphabetical order

• ls -l ~ | grep -v '^total' | tr -s ' ' | cut -d ' ' -f1,9 | sort -r -k2

• grep prints lines matching a pattern.
• -v stands for inverted match.
• The tr command replaces all occurrences of a character in a file, and print the result.
• -s replaces each input sequence of a repeated character that is listed in SET1 with a single occurrence of that character
• The command cut removes sections from each line of files
• -d use DELIM instead of TAB for field delimiter
• -f(number) select only these fields
• The sort command sort lines of text files
• -r reverse the result of comparisons
• -k start a key at POS1 (origin 1), end it at POS2 (default end of line).

Exercise 29

Use the command cal to find out on which day of the week your birthday is in 2050 and write your output to a newly created file

• ncal -m 6 2050 | grep ' 9 ' | cut -d ' ' -f1 > file
• cal -N -m 6 2050 | grep ' 9 ' | cut -d ' ' -f1 > file

• The command ncal displays a calendar and the date of Easter
• The command cal displays a calendar and the date of Easter
• -m displays the specified month
• The command cut removes sections from each line of files
• -d use DELIM instead of TAB for field delimiter
• -f(number) select only these fields
• '>' means overwrite a file

Exercise 30

Append the sentence 'THE END' to the file you created in the previous exercise without opening this file in an editor (hence: use a one-liner)

• echo 'THE END' >> file

# >> makes a file and saves it

Exercise 31

Create a subdirectory TEST in your homedir. Now create some files in it using this command line (and make sure you understand what happens!)

for foo in `seq 1 9`; do touch "file $RANDOM"; done && touch 'file keep me'

How can you remove all files except the file “file keep me” with just one 'oneliner'"

1) mkdir ~/TEST; cd ~/TEST
2) for foo in {1..9}; do touch "file $RANDOM"; done && touch 'file keep me'

• rm file\ [0-9]*
• ls file\ [0-9]* | while read file; do rm "$file";done
• ls -1 | grep -v 'file keep me' | xargs -d '\n' rm
• find . -type f ! -name 'file keep me' -delete

• The for keyword indicates a for loop ==> SYNTAX : for WORD [ in WORDLIST ... ] ; do ACTIONS ; done
• The command touch creates a new empty file(s) or change the times for existing file(s) to current time
• The rm command removes files or directories
• Remove (unlink) the FILE(s)
• The while command continuously executes the list list-2 as long as the last command in the list-1 returns an exit status of zero.
• The command xargs builds and executes command lines from standard input
• -d stands for delimiter ==> This can be used when the input consists of simply newline- separated items, although it is almost always better to design your program to use --null where this is possible.
• The find command searches for files in a directory hierarchy
• -f stands for regular file
• -delete stands for Delete files; true if removal succeeded.

Exercise 32

List the name and permissions of the first 10 directories in /etc.

Sample output: 
drwxr-xr-x /etc/ 
drwxr-xr-x /etc/alternatives 
drwxr-xr-x /etc/apache2 
drwxr-xr-x /etc/apache2/conf.d 
drwxr-xr-x /etc/apache2/mods-available 
drwxr-xr-x /etc/apache2/mods-enabled 
drwxr-xr-x /etc/apache2/sites-available 
drwxr-xr-x /etc/apache2/sites-enabled 
drwxr-xr-x /etc/apt d
rwxr-xr-x /etc/apt/apt.conf.d"

• ls -l /etc | head -10 | grep -v '^total' | awk '{print $1 " " $9}'

• The head command prints the first 10 lines of each FILE to standard output. With more than one FILE, precede each with aheader giving the file name. With no FILE, or when FILE is -, read standard input.
• grep prints lines matching a pattern
• -v stands for inverted match.
• awk '{print $(number)' prints the field with the matching number

Exercise 33

Same question as #32, but now also omit all error messages. "

• ls -l /etc 2>/dev/null | head -10 | grep -v '^total' | awk '{print $1 " " $9}'

• 2>/dev/null is used to redirect to a file
• The head command prints the first 10 lines of each FILE to standard output. With more than one FILE, precede each with aheader giving the file name. With no FILE, or when FILE is -, read standard input.
• grep prints lines matching a pattern.
• -v stands for inverted match.
• awk '{print $(number)' prints the field with the matching number

Exercise 34:

List the name and permissions of all files in the /etc directory containing the word 'host' in their name. Do this without the use of the commands grep and/or awk."

• find /etc/ -type f -name '*host*' 2>/dev/null | xargs -d '\n' ls -l # (recursive search)
• find /etc/ -maxdepth 1 -type f -name '*host*' 2>/dev/null | xargs -d '\n' ls -l
• ls -ld /etc/*host* | tr -s ' ' | cut -d ' ' -f1,9 # (also show files of type directory)   
• ls -ld /etc/*host* | sed -e '/^d.*/d' | tr -s ' ' | cut -d ' ' -f1,9

• The find command searches for files in a directory hierarchy
• -f stands for regular file
• -name stands for the name
• 2>/dev/null is used to redirect to a file
• The command xargs builds and executes command lines from standard input
• -d stands for delimiter ==> This can be used when the input consists of simply newline- separated items, although it is almost always better to design your program to use --null where this is possible.
• -maxdepth stands for the - levels of directories below the starting point
• The command cut removes sections from each line of files
• -d use DELIM instead of TAB for field delimiter
• -f(number) select only these fields
• The tr command replaces all occurrences of a character in a file, and print the result.
• -s replaces each input sequence of a repeated character that is listed in SET1 with a single occurrence of that character

LABO 1

In order to connect to a remote SSH server leia.uclllabs.be in this example, execute the following command

'user:~$' • ssh r1234567@leia.uclllabs.be -p 22345

The client configuration file can make your life easier because default command line options could be specified in this configuration file

Normally you would use the following command to log in to leia

'user:~$' • ssh -p 22345 r1234567@leia.uclllabs.be  

"FILE: ~/.ssh/config"
    Host leia
    HostName leia.uclllabs.be
    Port 22345
    User r1234567

The following command creates an ssh tunnel. Local tcp port CPORT (on your laptop) will be tunneled through GATEWAY to SERVER port

'user:~$' • ssh -f GWUSERNAME@GATEWAY -L CPORT:SERVER:SPORT -N

• The -f argument instructs the ssh instance to go into the background, and -N instructs it to not launch a shell.

• The -L argument specifies that the given port on the local (client) host is to be forwarded to the given host and port on the remote side.

Try it yourself: There is a website running - https://darthvader.uclllabs.be/nw2/lab1 which is only accessible from 193.191.177.1. That's the IP address of leia.uclllabs.be.

'user:~$' • ifconfig | grep -B1 inet

• ifconfig configures a network interface
• The grep command grep searches the named input FILEs
• -B NUM prints NUM lines of leading context before matching lines

With this command, you can create an ssh tunnel between server leia.uclllabs.be and the webserver darthvader.uclllabs.be.

'user:~$' • ssh -p 22345 -f r0123456@leia.uclllabs.be -L 10000:darthvader.uclllabs.be:443 -N

• The -p argument shows which port to connect to on the remote host
• The -f argument instructs the ssh instance to go into the background, and -N instructs it to not launch a shell.
• The -L argument specifies that the given port on the local (client) host is to be forwarded to the given host and port on the remote side.

First find the ssh process to close:

'user:~$' • ps fauxw | grep darthvader

• The ps command reports a snapshot of the current processes
• The f argument does a full-format listing
• The a argument selects all processes except both session leaders and processes not associated with a terminal.
• The u argument selects by effective user ID (EUID) or name.
• The x arugment Lift the BSD-style "must have a tty" restriction, which is imposed upon the set of all processes when some BSD-style (without "-") options are used or when the ps personality setting is BSD-like.
• The w argument gives wide output
• grep searches the named input FILEs

Next send a sigterm signal to the correct pid (process identifier) = second column in ps fauxw output

'user:~$' • kill 12345

And if you need to close all running ssh sessions/tunnels use the following command:

'user:~$' • killall ssh

By default kill sends a SIGTERM signal (similar to clicking on the X to close a window (Chrome, Notepad, Gnome-terminal,..). If you need to force close an application, send the SIGKILL signal:

'user:~$' • kill -9 12345
'user:~$' • killall -9 ssh

Use a text based browser on leia to verify the remote website is accessible through the ssh tunnel.

'user:~$' • lynx https://localhost:10000/nw2/lab1-all-but-leia
'user:~$' • curl -k -s https://localhost:10000/nw2/lab1-all-but-leia/ | html2text

• The argument -k instructs curl not to verify the server certificate.
• The argument -s makes curl mute make sure that it doesn't show progress meter or error messages

OpenSSH has built-in support for dynamic port forwarding via the SOCKS proxy protocol. This command turns your SSH client (on your laptop) into a SOCKS proxy server: "

'user:~$' • ssh -f user@GATEWAY -D 10000 -N -p port

• The -f argument instructs the ssh instance to go into the background, and -N instructs it to not launch a shell.
• The -p argument shows which port to connect to on the remote host
• The -D argument specifies a local “dynamic” application-level port forwarding

There are 3 different levels of debug modes that can help troubleshooting issues. With the -v option SSH prints debugging messages about its progress.

This is helpful in debugging connection, authentication, and configuration problems. Multiple -v options increase the verbosity. Maximum verbosity is three levels deep.

'user:~$' • ssh leia -v
'user:~$' • ssh leia -vv
'user:~$' • ssh leia -vvv

This command will aid you in finding an available tcp port: "

'user:~$' • netstat -lnt | grep -oP ':\K[^:][0-9]+' | sort -un

• The command netstat prints network connections, routing tables, interface statistics, masquerade connections, and multicast memberships
• The -l shows only listening sockets.
• The -n shows numerical addresses instead of trying to determine symbolic host, port or user names.
• The -t argument shows only TCP ports
• The command grep searches for PATTERNS in each FILE.
• The -o argument prints only the matched (non-empty) parts of a matching line, with each such part on a separate output line.
• The -p argument Interpret I as Perl-compatible regular expressions (PCREs).
• : matches the character : literally (case sensitive)
• \K resets the starting point of the reported match.
• [^:] matches a single character not present in the list below
• [0-9] matches a single character present in the list below
• '+' matches the previous token between one and unlimited times, as many times as possible, giving back as needed (greedy)
• The command sort sorts lines of text files
• The -u argument output only the first of an equal run
• The -n argument compares according to string numerical value

For instance, we can scan all ports up to 1000 by issuing this command:

'user:~$' • nc -vz leia.uclllabs.be 1-1000

• The command nc is a TCP/IP swiss army knife
• The -v argument is for verbose
• The -z argument is for zero-I/O mode [used for scanning]

"Messages returned by Netcat are sent to standard error. You can send the standard error messages to standard out, which will allow filtering the results. Stderr can be redirected to stdout via the 2>&1 bash syntax. Then use grep to filter

Alternatively you could use |& as a shorthand for 2>&1 |. "

'user:~$' • nc -z -n -v 193.191.177.1 1-1000 2>&1 | grep succeeded
'user:~$' • nc -z -n -v 193.191.177.1 1-1000 &| grep succeeded

• The command nc is a TCP/IP swiss army knife
• The -z argument is for zero-I/O mode [used for scanning]
• The -v argument is for verbose
• The -v argument is used for short version

Netcat is a very slow TCP port scanner as it will wait for a TCP timeout for every port it tries. Add option -w 1 to increase scanning speed, and execute this command on leia to circumvent slowdown due to firewalled ports.

Netcat isn't designed to scan for open ports, it's functionality is very limited. If you need to scan for open ports, use a real portscanner like Nmap"

'user:~$' • nmap -p1-1000 193.191.177.1

• The command nmap is a network exploration tool and security / port scanner
• The -p argument specifies which ports you want to scan

Once again, you need to choose one end of the connection to listen for connections.

However, instead of printing information onto the screen, you will place all information straight into a file

'user:~$' • nc -l -p 10000 > received_file

• The command nc is a TCP/IP swiss army knife
• The -l argument is used as listening mode
• The -p argument specifies which ports you want to scan
• The > overwrites a file

On the second computer, create a simple text file by typing: "

'user:~$' • echo "Hello, this is a file" > original_file

You can now use this file as an input for the Netcat connection you will establish to the remote listening computer.

The file will be transmitted just as if you had typed it interactively: "

'user:~$' • nc -l -p 10000 < original_file

On the receiving end, you can anticipate a file coming over that will need to be unzipped and extracted by typing: "

'user:~$' • nc -l -p 10000 | tar xzvf -

• The command nc is a TCP/IP swiss army knife
• The -l argument is used as listening mode
• The -p argument specifies which ports you want to scan
• The tar command is a GNU version of the tar archiving utility
• The -x argument extracts files from an archive
• The -z argument unzips the file
• The -v argument is for verbose
• The -f argument uses archive file
• The ending dash (-) means that tar will operate on standard input, which is being piped from Netcat across the network when a connection is made.

On the side with the directory contents you want to transfer, you can pack them into a tarball and then send them to the remote computer through Netcat:

'user:~$' • tar -czf - * | nc leia.uclllabs.be 10000

• The tar command is a GNU version of the tar archiving utility
• The -c argument creates a new archive
• The -z argument unzips the file
• The -f argument uses archive file
• The command nc is a TCP/IP swiss army knife

In the following example, we create a full backup of our home directory on leia (all files and folders) using tar - Netcat method.

Execute this on leia: "

'user:~$' • tar -cz ~ | nc -l -p 10000

• The tar command is a GNU version of the tar archiving utility
• The -c argument creates a new archive
• The -z argument unzips the file
• The command nc is a TCP/IP swiss army knife
• The -l argument is used as listening mode
• The -p argument specifies which ports you want to scan

And this on your laptop (the machine receiving backups):

'user:~$' • cd /path/to/where_I_want_to_store_my_backup
'user:~$' • nc leia.uclllabs.be 10000 | tar -xzf -

• The command nc is a TCP/IP swiss army knife
• The tar command is a GNU version of the tar archiving utility
• The -x argument extracts files from an archive
• The -z argument unzips the file
• The -f argument uses archive file w

This is just one example of transferring more complex data from one computer to another. Another common idea is to use the dd command to image a disk on one side and transfer it to a remote computer.

'user@leia:~$' • tar -cz ~ | pv | nc -l -p 10000

'user@laptop:~$' • nc leia.uclllabs.be 10000 | pv > backup.tar.gz

• The tar command is a GNU version of the tar archiving utility
• The command nc is a TCP/IP swiss army knife
• The -c argument creates a new archive
• The pv command monitors the progress of data through a pipe
• The -z argument unzips the file
• The > argument overwrites the file
• The pv command monitors the progress of data through a pipe
• The command nc is a TCP/IP swiss army knife
• The -l argument is used as listening mode
• The -p argument specifies which ports you want to scan

'user@leia:~$' • tar -cz ~ | pv | nc '-N' -l -p 10000

'user@laptop:~$' • nc -d leia.uclllabs.be 10000 | pv > backup.tar.gz

• The tar command is a GNU version of the tar archiving utility
• The command nc is a TCP/IP swiss army knife
• The -c argument creates a new archive
• The -d argument makes sure that it does not attempt to read from stdin.
• The -z argument unzips the file

'user@laptop:~$' • nc leia.uclllabs.be 10000 </dev/null | pv | tar xzf -

• The pv command monitors the progress of data through a pipe
• The -N argument shuts the network socket down after EOF on the input
• The -l argument is used as listening mode
• The -p argument specifies which ports you want to scan
• /dev/null is for error log

Exercise 1:

How do you verify if the packages openssh-server and openssh-client are installed on your Debian system? (Hint: dpkg -l ...)"

'user:~$' • dpkg -l openssh-server
'user:~$' • dpkg -l openssh-client

• The argument -l is used for list

Exercise 2:

How do you verify if the openssh-server is running?

'user:~$' • systemctl status ssh 

Exercise 3:

How to check which port is used by the SSH server daemon? (Hint: netstat, ss or lsof)

'root #' • netstat -lntp | grep sshd
'root #' • ss -lntp | grep sshd
'root #' • lsof -i tcp | awk '$1=="sshd" && $10=="(LISTEN)" {print $9}' | cut -d ':' -f2 | sort -u

• The netstat command prints network connections, routing tables, interface statistics, masquerade connections, and multicast memberships
• The -l argument is used as listening mode
• The -n argument shows numerical addresses instead of trying to determine symbolic host, port or user names.
• The -p argument shows the PID and name of the program to which each socket belongs
• The grep command prints lines matching a pattern
• The ss command is another utility to investigate sockets
• The lsof command lists open files
• The -i argument selects the listing of files any of whose Internet address matches the address specified in i.
• The awk command is used for pattern scanning and processing language
• The cut command removes sections from each line of files
• The -d argument use DELIM instead of TAB for field delimiter
• The -f argument selects only these fields
• The sort command sort lines of text files
• The -u argument outputs only the first of an equal run

Exercise 4:

How do you disconnect from a remote SSH session?

'user:~$' • exit
'user:~$' • logout
'user:~$' • <CTRL>+d

Exercise 5:

Is there a config file for the OpenSSH server? If so, where is it located in the file system tree?

Yes: '/etc/ssh/sshd_config'

Exercise 6:

Create and demonstrate a simple web server with Netcat.

'user:~$' • while true; do { echo -e "HTTP/1.1 200 OK\r\n$(date)\r\n\r\n<h1>hello world from $(hostname) on $(date)</h1>" | nc -vl -p 10000; } done

• The while command The while command continuously executes the list list-2 as long as the last command in the list list-1 returns an exit status of zero.
• The echo command displays a line of text
• The -e argument enables interpretation of backslash escapes
• The command nc is a TCP/IP swiss army knife
• The -v argument uses verbose
• The -l argument is used for listen mode, for inbound connects
• The -p argument is used for local port number

LABO 2

What is the IP address of your computer?

What is the status code returned from the server to your browser?

When was the HTML file that you are retrieving last modified on the server?

'user:~$' • echo -ne 'HEAD /HTTP-Wireshark-file1.html HTTP/1.1\r\nHost:  virtualhostname.x.cnw2.uclllabs.be\r\n\r\n' | nc localhost 80 | grep 'Last-Modified:'

'user:~$' • tshark -r http.pcapng -Y http -T fields -e http.last_modified

• The -n argument does not output the trailing newline
• The -e argument enables interpretation of backslash escapes
• The nc command is a TCP/IP swiss army knife
• The -r argument reads the packet date from infile
• The -Y command captures the link type
• The -e argument (in tshark command) adds a field to the list of fields to display if -T fields is selected

How many bytes of content are being returned to your browser?

What software and version is the web server running?

'user:~$' • tshark -r http.pcapng -Y http.server -T fields -e ip.src -e http.server | sort -u

• The -r argument reads the packet date from infile
• The -Y argument captures the link type
• The -T argument sets the format of the output when viewing decoded packet data.
• The -e argument (in tshark command) adds a field to the list of fields to display if -T fields is selected
• The sort command sorts lines of text files
• The -u argument output only the first of an equal run

Explain in detail the above tshark command.

What TCP ports are in use at the client and the server during your browsing session?

'user:~$' • tshark -r http.pcapng -Y http -T fields -e tcp.port | sort -u

• The -r argument reads the packet date from infile
• The -Y argument captures the link type
• The -T argument sets the format of the output when viewing decoded packet data.
• The -e argument (in tshark command) adds a field to the list of fields to display if -T fields is selected
• The sort command sorts lines of text files
• The -u argument output only the first of an equal run

Exercise 1:

Which HTTP method was used the most during the entire browsing session?

'user:~$' • tshark -r http.pcapng -Y http.request.method -T fields -e http.request.method | sort | uniq -c | head -1 | awk '{print $2}'
'user:~$' • tshark -r http.pcapng -Y http.request.method -T fields -e http.request.method | sort | uniq -c | awk 'NR=1{print $2}'

• The tshark command dumps and analyzes network traffic
• The -r argument reads the packet date from infile
• The -T argument sets the format of the output when viewing decoded packet data.
• The -e argument (in tshark command) adds a field to the list of fields to display if -T fields is selected
• The sort command sorts lines of text files
• The uniq command reports or omits repeated lines
• The -c argument prefixes lines by the number of occurences
• The head command shows output for only the first part of files
• The awk command is used for pattern scanning and processing language

In case you would like to automate this: With tshark and a Bash loop"

'user:~$' • tshark -r http.pcapng -Y 'http.request.method==GET' -T fields -e tcp.srcport | sort -u | while read PORT;do tshark -r http.pcapng -Y "tcp.dstport==$PORT && http.server contains Apache" -T fields -e ip.src;done | sort -u

• The tshark command dumps and analyzes network traffic
• The -r argument reads the packet date from infile
• The -T argument sets the format of the output when viewing decoded packet data.
• The -e argument (in tshark command) adds a field to the list of fields to display if -T fields is selected
• The sort command sorts lines of text files
• The -u argument output only the first of an equal run
• The -Y command captures the link type

Exercise 2:

How many HTTP GET request messages did your browser send?

'user:~$' • tshark -r http.pcapng -Y http.request.method==GET | wc -l

• The tshark command dumps and analyzes network traffic
• The -r argument reads the packet date from infile
• The wc command prints a newline, word, and byte counts for each file
• The -l argument prints the newline counts

To which Internet addresses were these GET requests sent?

'user:~$' • tshark -r http.pcapng -Y http.request.method==GET -T fields -e ip.dst | sort -u

• The tshark command dumps and analyzes network traffic
• The -r argument reads the packet date from infile
• The -Y command captures the link type
• The -T argument sets the format of the output when viewing decoded packet data.
• The -e argument (in tshark command) adds a field to the list of fields to display if -T fields is selected
• The sort command sorts lines of text files
• The -u argument output only the first of an equal run

Exercise 5:

Use Netcat to download these images. check the echo -ne options or use printf. If needed, slow down netcat with option -i. The image part in the HTTP stream starts after a blank line.

'user:~$' • echo -ne "GET /nw2/images/image1.jpg HTTP/1.1\r\nHost: darthvader.uclllabs.be\r\n\r\n" |\ nc darthvader.uclllabs.be 80 | sed '1,/^\r/d' > image1.jpg

'user:~$' • echo -ne "GET /nw2/images/image1.jpg HTTP/1.1\r\nHost: darthvader.uclllabs.be\r\n\r\n" |\ nc darthvader.uclllabs.be 80 | grep -A9999999999999999 -B0 -Pa 'JFIF' > image1.jpg

• The -n argument does not output the trailing newline
• The -e argument enables interpretation of backslash escapes
• The sed command is a stream editor for filtering and transforming text
• The nc command is a TCP/IP swiss army knife
• The -A argument prints NUM lines of trailing context after matching lines.
• The -B argument interprets PATTERN as a Perl regular expression (PCRE, see below).
• The -a argument processes a binary file as if it were text; this is equivalent to the --binary-files=text option.

Exercise 7:

Use httpie, a cURL-like tool for humans to inspect the various HTTP headers in request and responses. Connect to various websites and explain the use of the HTTP headers.

'user:~$' • http -v -a Rey:StarWars http://darthvader.uclllabs.be/nw2/private/

• The -v argument is for verbose

Exercise 8:

A simulated phone is running at http://darthvader.uclllabs.be/nw2/phone/. Create a oneliner to bruteforce the pincode. Tip: pincode range: 1200-1300

'user:~$' • for foo in {1200..1300}; do if wget -q --http-user='admin' --http-password=$foo http://darthvader.uclllabs.be/nw2/phone; then echo $foo;break;fi;done

• The wget command is the non-interactive network downloader
• The -q argument turns of the wget's output
• The --http-user AND --http-password specifies the username and the password on a http server

Exercise 9:

"Put the following text.txt on your web server. This text contains the string Goed bezig :-)

Write an HTTP request by using the Range header so your web server will only return this exact string 'Goed bezig :-)'. Try to do this by only using netcat

'user:~$' • curl http://your.server.name/output.txt -i -H "Range: bytes=1-"
'user:~$' • echo -ne "GET /output.txt HTTP/1.1\r\nHost: your.server.name\r\nRange: bytes=1-\r\n\r\n" | nc your.server.name 80

• The curl command is used to transfer a URL
• The -i argument includes the HTTP-header in the output
• The -H argument is used as a extra header to use when getting a web page
• The nc command is a TCP/IP swiss army knife
• The -n argument does not output the trailing newline
• The -e argument enables interpretation of backslash escapes

Exercise 10:

This can be accomplished by sending the output of tshark or tcpdump to STDOUT instead of a regular file. Direct this STDOUT stream to Wireshark running on your local computer.

'root #' • ssh myserver.X.cnw2.uclllabs.be tcpdump -nli eth0 not tcp port 22345 -s0 -w - | wireshark -nki -

'root #' • ssh myserver.X.cnw2.uclllabs.be 'tshark -nli eth0 -f "not tcp port 22345" -s0 -w -' | wireshark -nki -

• The ssh command is a remote login program
• The -n argument redirects stdin from /dev/null (actually, prevents reading from stdin).
• The -l argument specifies the user to log in as on the remote machine.
• The -i argument selects a file from which the identity (private key) for public key authentication is read.
• The -s argument may be used to request invocation of a subsystem on the remote system
• The -w argument Requests tunnel device forwarding with the specified tun(4) devices between the client (local_tun) and the server (remote_tun).
• The -n argument disables network object name resolution (such as hostname, TCP and UDP port names), the -N flag might override this one.
• The -k argument starts the capture session immediately.
• The -i argument sets the name of the network interface or pipe to use for live packet capture.
• The -f argument (in tshark command) sets the capture filter expression

Exercise 11:

Capture some HTTP traffic while browsing several websites and save it to the file http.pcapng.

You can also use the test capture in /home/logs on leia. create a CLI oneliner which parses the captured file http.pcapng and displays all HTTP server strings which do not contain Apache.

Only the commands tshark and sort are allowed.

'user:~$' • tshark -r http.pcapng -Y 'http.server && !(http.server contains Apache)' -T fields -e http.server | sort -u

• The -r argument reads the packet date from infile
• The -Y command captures the link type
• The -T argument sets the format of the output when viewing decoded packet data.
• The -e argument (in tshark command) adds a field to the list of fields to display if -T fields is selected
• The sort command sorts lines of text files
• The -u argument output only the first of an equal run

Exercise 12:

This exercise is a small variation of the previous one. Count and sort all HTTP server strings which do not contain Apache in HTTP responses on your GET requests.

'user:~$' • tshark -r http.pcapng -Y '!(http.request.method==GET)' -T fields -e tcp.srcport | sort -u | while read PORT;do tshark -r http.pcapng -Y "tcp.dstport==$PORT && http.server && !(http.server contains Apache)" -T fields -e http.server;done | sort | uniq -c | sort -rn

• The tshark command dumps and analyzes network traffic
• The -r argument reads the packet date from infile
• The -Y command captures the link type
• The -e argument (in tshark command) adds a field to the list of fields to display if -T fields is selected
• The sort command sorts lines of text files
• The -u argument output only the first of an equal run
• The -T argument sets the format of the output when viewing decoded packet data.
• The uniq command reports or omits repeated lines
• The -c command prefixes lines by the number of occurrences
• The -r argument (in sort command) reverses the results of comparisons
• The -n compare according to string numerical value

Labo 3

Match one or more d characters: d dd ddd dddd "

'user@:~$' • grep -P 'd+' file

'user@:~$' • cat file | grep -P 'd+'

• d matches the character d literally (case sensitive)
• The grep command prints lines matching a pattern
• The -P argument interprets PATTERN as a Perl regular expression

Color and colour are different spellings of the same word.

Color is the preferred spelling in American English, and colour is preferred in all other main varieties of English. The following regex matches both. "

'user@:~$' • grep -P 'colou?r' file

• colo matches the characters colo literally (case sensitive)
• u matches the character u literally (case sensitive)
• ? matches the previous token between zero and one times, as many times as possible, giving back as needed (greedy)
• r matches the characters r literally (case sensitive)

Match wh following by one, two or three e character: 'whe whee wheee'

'user@:~$' • grep -P 'whe{1,3}' file

• The grep command prints lines matching a pattern
• The -P argument interprets PATTERN as a Perl regular expression
• whe matches the characters whe literally (case sensitive)
• {1,3} matches the previous token between 1 and 3 times, as many times as possible, giving back as needed (greedy)

Match either gray or grey

'user@:~$' • grep -P '{gray|grey}' file

• The grep command prints lines matching a pattern
• The -P argument interprets PATTERN as a Perl regular expression
• '{gray|grey}' must match gray or gray

Let's say you want to match a text like !abc! or !123!. Only these two are possible, and you want to capture the abc or 123.

'user@:~$' • grep -P '!(abc|123)!' file

• The grep command prints lines matching a pattern
• The -P argument interprets PATTERN as a Perl regular expression
• '{gray|grey}' must match !abc! or !123!

Search /usr/share/dict/dutch for all words which end in 'vuur', without matching the word 'vuur' itself.

'user@:~$' • grep -P '^.+vuur$' /usr/share/dict/dutch

• ^ asserts position at start of a line
• . matches any character (except for line terminators)
• matches the previous token between one and unlimited times, as many times as possible, giving back as needed (greedy)
• vuur matches the characters vuur literally

List all words which contain 32 or more letters.

user@:~$' • grep -P '^.{32}$' /usr/share/dict/dutch

• ^ asserts position at start of a line
• . matches any character (except for line terminators)
• {32} matches the previous token exactly 32 times
• $ asserts position at the end of a line

List all words starting with the letter 'b', ending with the letter 'l' which are exactly 4 letters long. Capitalize the output

'user@:~$' • grep -E '^b.{2}l$' /usr/share/dict/dutch | tr [:lower:] [:upper:]
    
'user@:~$' • perl -ne 'print uc if /^b.{2}l$/' /usr/share/dict/dutch

• ^ asserts position at start of a line
• b matches the character b literally (case sensitive)
• . matches any character (except for line terminators)
• {2} matches the previous token exactly 2 times
• l matches the character l literally (case sensitive)
• $ asserts position at the end of a line
• The perl command is how to execute the Perl interpreter
• The -n argument causes Perl to assume the following loop around your program, which makes it iterate over filename arguments somewhat like sed -n or awk:
• The -e argument may be used to enter one line of program.

List all palindromes with exactly 4 characters

```
'user@:~$' • grep -P '^([A-Za-z])([A-Za-z])\2\1$' /usr/share/dict/dutch

'user@:~$' • perl -lne 'print if $_ eq reverse && length($_) eq 4' /usr/share/dict/dutch
```

• A-Z matches a single character in the range between A and Z (case sensitive)
• a-z matches a single character in the range between a and z (case sensitive)
• \2 matches the same text as most recently matched by the 2nd capturing group
• \1 matches the same text as most recently matched by the 1st capturing group
• $ asserts position at the end of a line

Exercise 1:

List all words out of the file /usr/share/dict/dutch which contain 4 or 5 consecutive vowels.

```
'user@:~$' • cat /usr/share/dict/dutch | grep -P '[aeiou]{4,5}'
```

• aeiou matches a single character in the list aeiou (case sensitive)
• {4,5} matches the previous token between 4 and 5 times, as many times as possible, giving back as needed (greedy)

Exercise 2:

Count the number of words out of the file /usr/share/dict/dutch which which start with 'ver', and end with 'en'. ('verplaatsen' is ok, 'overkomen' and 'vertrek' are not ok)

```
'user@:~$' • grep -P '^ver.*en$' /usr/share/dict/dutch | wc -l
```

• ^ asserts position at start of a line
• ver matches the characters ver literally (case sensitive)
• . matches any character (except for line terminators)
• '*' matches the previous token between zero and unlimited times, as many times as possible, giving back as needed (greedy)
• $ asserts position at the end of a line
• The wc command prints newline, word, and byte counts for each file
• The -l argument prints the newline counts

Exercise 3:

In those annoying night television games people must guess words. Given are all the letters the word consist of and a list of dots, one for every letter.

E.g. We are looking for a 23 letter word with a 'v' in position 8: '.......v...............'. Use the letters 'enrtiscau' for the other positions."

```
'user@:~$' • cat /usr/share/dict/dutch | grep -P '[enrtiscau]{7}v[enrtiscau]{15}'
```

• enrtiscau matches a single character in the list enrtiscau (case sensitive)
• {7} matches the previous token exactly 7 times
• {15} matches the previous token exactly 15 times

Exercise 4:

Show 'System load = X Y Z' replacing X, Y and Z by the values displayed by the uptime command.

```
'user@:~$' • echo "System load =" $(uptime | awk -F': ' '{print $2}'| tr -d ',')
```

Exercise 5:

List the usernames of the students who logged in from outside of the UCLL network.

```
'user@:~$' • who | grep -Pv '(.*)\s+pts.*\s+\(10\.|tmux' | awk '/^r[0-9]/{print $1}' | sort -u
```

• . matches any character (except for line terminators)
• '*' matches the previous token between zero and unlimited times, as many times as possible, giving back as needed (greedy)
• \s matches any whitespace character (equivalent to [\r\n\t\f\v ])
• r matches the character r literally (case sensitive)
• • matches the previous token between one and unlimited times, as many times as possible, giving back as needed (greedy)
• tmux matches the characters tmux literally (case sensitive)
• ^ asserts position at start of a line
• 0-9 matches a single character in the range between 0 and 9 (case sensitive)

Exercise 6:

How many times have students logged into leia from outside the UCLL network?

```
'user@:~$' • last | grep -Pv '.*?pts.*?\s+10\.' | awk '/^r[0-9]/{print $1}' | wc -l
```

• . matches any character (except for line terminators)
• *? matches the previous token between zero and unlimited times, as few times as possible, expanding as needed (lazy)
• pts matches the characters pts literally (case sensitive)
• r matches the character r literally (case sensitive)
• \s matches any whitespace character (equivalent to [\r\n\t\f\v ])
• '+' matches the previous token between one and unlimited times, as many times as possible, giving back as needed (greedy)
• ^ asserts position at start of a line

Exercise 7:

Show the file /etc/debconf.conf on screen without comment lines (i.e. lines starting with a #)

```
'user@:~$' • cat /etc/debconf.conf | grep -vP '^#'
```

• ^ asserts position at start of a line
• '#' matches the character # literally (case sensitive)

Exercise 8:

List all unique IP addresses that contacted your Apache web server.

```
'user@:~$' • cat /var/log/apache2/wiki-ssl-access.log | awk '{print $1}' | sort -u
```

Exercise 9:

List all unique IP addresses that contacted the ssh daemon of your Apache web server.

```
'user@:~$' • cat /var/log/auth.log | grep -oP 'sshd.*?\K[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | sort -u
```

• sshd matches the characters sshd literally
• . matches any character (except for line terminators)
• *? matches the previous token between zero and unlimited times, as few times as possible, expanding as needed (lazy)
• \K resets the starting point of the reported match. Any previously consumed characters are no longer included in the final match
• [0-9]{1,3} matches the previous token between 1 and 3 times, as many times as possible, giving back as needed (greedy)
• 0-9 matches a single character in the range between 0 and 9 (case sensitive)
• . matches the character . literally (case sensitive)

Exercise 10:

Create a regular expression to match all words in the dictionary /usr/share/dict/dutch which with 10 unique letters.

```
'user@:~$' • cat /usr/share/dict/dutch | grep -P '^.{10}$' | grep -vP '(.).*\1'
```

• ^ asserts position at start of a line
• . matches any character (except for line terminators)
• {10} matches the previous token exactly 10 times
• '*' matches the previous token between zero and unlimited times, as many times as possible, giving back as needed (greedy)
• \1 matches the same text as most recently matched by the 1st capturing group

Exercise 11:

To combat spam, a mail server administrator wants to reject mail coming from home users. The IP addresses home users always seem to connect from have hostnames like this: ip.domain. Create a regex which matches all these host names.

```
'user@:~$' • cat mail.log | grep -oP 'NOQUEUE: reject: RCPT from \K[0-9]{1,3}.*?\[' | tr -d '['
```

• [0-9]{1,3} matches the previous token between 1 and 3 times, as many times as possible, giving back as needed (greedy)
• . matches any character (except for line terminators)
• *? matches the previous token between zero and unlimited times, as few times as possible, expanding as needed (lazy)

Exercise 12:

List all unique firefox 2.0.0.x versions used to connect to www.khleuven.be. Use log file /home/logs/apache_google.log. "

```
'user@:~$' • cat apache_google.log | grep -oP 'Firefox\/2\.0\.0\.[0-9]+' | sort -n -t '.' -k4 -u
```

• Firefox matches the characters Firefox literally (case sensitive)
• / matches the character / literally (case sensitive)
• 2 matches the character 2 literally (case sensitive)
• . matches the character . literally (case sensitive)
• 0 matches the character 0 literally (case sensitive)
• [0-9]+ matches the previous token between one and unlimited times, as many times as possible, giving back as needed (greedy)

Exercise 13:

List all words with 14, 15 or 16 unique letters.

```
'user@:~$' • for foo in 14 15 16; do echo "Words with $foo letters:" $(echo "grep -vP '(.).*\1' /usr/share/dict/dutch | grep -P '^.{$foo}$'"| sh);done

'user@:~$' • for foo in 14 15 16; do echo "Words with $foo letters:" $(grep -vP '(.).*\1' /usr/share/dict/dutch | grep -P "^.{$foo}$");done

'user@:~$' • for foo in 14 15 16; do echo "Words with $foo letters:" && echo "grep -vP '(.).*\1' /usr/share/dict/dutch | grep -P '^.{$foo}$'"| sh;done

'user@:~$' • for foo in 14 15 16; do echo "Words with $foo letters:" && grep -vP '(.).*\1' /usr/share/dict/dutch | grep -P "^.{$foo}$";done
```

• . matches any character (except for line terminators)
• '*' matches the previous token between zero and unlimited times, as many times as possible, giving back as needed (greedy)
• \1 matches the same text as most recently matched by the 1st capturing group
• ^ asserts position at start of a line
• $ asserts position at the end of a line
• foo} matches the characters foo} literally (case sensitive) $ asserts position at the end of a line