HTB Academy - Filter Contents

🎯 Objective

Quick, repeatable techniques to read, reshape, and extract data from files and web pages using core Linux tools (head, tail, cut, column, awk, sed) plus a couple of network/HTTP one‑liners.

✅ TL;DR

• Peek file edges: head, tail
• Slice columns: cut -d ":" -f <fields>
• Tabular view: column -t
• Pick fields / logic: awk '{print $1, $NF}'
• Search/replace: sed 's/old/new/g'
• Count listening services (IPv4, non‑localhost): ss -l -4 | ... | wc -l
• Scrape unique site paths: curl | grep -Eo ... | sort -u | wc -l

🧰 Prerequisites

• Linux shell (bash/zsh)
• Packages typically preinstalled on most distros:
  • coreutils (for head, tail)
  • grep, awk, sed
  • util-linux (often provides column)
  • iproute2 (for ss)
  • curl

1) head & tail: peek the beginning / end of a file

Sometimes you only need the first or last few lines of a file.

First 10 lines (default):

head /etc/passwd

Last 10 lines (default):

tail /etc/passwd

Tip: Customize counts with -n, e.g., head -n 20 file or tail -n 50 file.

Example outputs (from your notes):

# head /etc/passwd (excerpt)
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
...

# tail /etc/passwd (excerpt)
miredo:x:115:65534::/var/run/miredo:/usr/sbin/nologin
usbmux:x:116:46:usbmux daemon,,,:/var/lib/usbmux:/usr/sbin/nologin
...
user6:x:1000:1000:,,,:/home/user6:/bin/bash

2) cut: extract delimited fields

When data is delimited (e.g., colon‑separated), use cut to select fields.

List only shell users (exclude false/nologin), show usernames (field 1):

cat /etc/passwd \
  | grep -v "false\|nologin" \
  | cut -d ":" -f1

Example result (from your notes):

root
sync
mrb3n
cry0l1t3
htb-student

Flags:

• -d ":" → delimiter is a colon
• -f1 → print only the first field

3) column: make clean tables

column -t aligns whitespace‑separated data into a tidy table.

Turn /etc/passwd into a quick table (replace : → space, then tabulate):

cat /etc/passwd \
  | grep -v "false\|nologin" \
  | tr ":" " " \
  | column -t

Example look (from your notes):

root        x  0     0     root       /root            /bin/bash
sync        x  4     65534  sync       /bin             /bin/sync
mrb3n       x  1000  1000   mrb3n      /home/mrb3n      /bin/bash
cry0l1t3    x  1001  1001             /home/cry0l1t3    /bin/bash
htb-student x  1002  1002             /home/htb-student /bin/bash

4) awk: pick fields smartly

awk is great for logic and field selection. $1 is the first field; $NF is Number of Fields → the last field.

Print username and shell (first & last fields):

cat /etc/passwd \
  | grep -v "false\|nologin" \
  | tr ":" " " \
  | awk '{print $1, $NF}'

Example output (from your notes):

root /bin/bash
sync /bin/sync
mrb3n /bin/bash
cry0l1t3 /bin/bash
htb-student /bin/bash

5) sed: search & replace at scale

Replace bin with HTB everywhere in the last column output:

cat /etc/passwd \
  | grep -v "false\|nologin" \
  | tr ":" " " \
  | awk '{print $1, $NF}' \
  | sed 's/bin/HTB/g'

Example output (from your notes):

root /HTB/bash
sync /HTB/sync
mrb3n /HTB/bash
cry0l1t3 /HTB/bash
htb-student /HTB/bash

s/old/new/g → substitute old with new globally on each line.

6) Count listening services (IPv4, non‑localhost)

Question: How many services are listening on all interfaces? (IPv4 only, exclude 127.0.0.1)

One‑liner:

ss -l -4 | grep -v "127\.0\.0" | grep "LISTEN" | wc -l

Flags explained:

• -l — listening sockets only
• -4 — IPv4 only
• grep -v "127\.0\.0" — remove localhost listeners
• grep "LISTEN" — only listening state
• wc -l — count them

Tip: Add sudo if you suspect you’re missing privileged sockets.

7) Scrape unique paths from a site with curl + grep

Task: Use cURL from Pwnbox (not the target machine) to fetch the page HTML for https://www.inlanefreight.com, extract unique paths for that domain, and return the count.

Your pipeline (counts unique URLs found in HTML):

curl https://www.inlanefreight.com \
  | grep -Eo "(http|https)://[a-zA-Z0-9./?=_%:-]*" \
  | sort -u \
  | wc -l

Alternative (restrict strictly to inlanefreight.com domain and count unique paths):

curl -s https://www.inlanefreight.com \
  | grep -Eo 'https?://(www\.)?inlanefreight\.com(/[a-zA-Z0-9._/?=:%-]*)?' \
  | sed -E 's#https?://(www\.)?inlanefreight\.com##' \
  | sed 's#//#/#g' \
  | sort -u \
  | wc -l

• First grep limits matches to that domain.
• First sed strips scheme + host → leaves only the path.
• sort -u dedupes paths, wc -l counts them.

🧪 Quick Self‑Checks

• Can you print only the 3rd and 7th fields from a colon‑delimited file?
  bash Copy

  cut -d ":" -f3,7 /etc/passwd | column -t -s :

• Can you show the top 20 and bottom 20 lines of a file quickly?
  bash Copy

  head -n 20 file && echo "---" && tail -n 20 file

• Can you list all users whose shell is /bin/bash?
  bash Copy

  awk -F: '$NF=="/bin/bash"{print $1}' /etc/passwd

🧭 Troubleshooting Notes

• No column found? Try sudo apt-get install bsdmainutils (older) or use util-linux package.
• ss shows nothing? Try with sudo or check iptables/nftables and systemd socket units.
• grep -E errors? On BusyBox, use egrep or ensure -E is supported.

📌 Cheat Sheet

# head / tail
head -n 15 file.txt
tail -n 50 file.txt

# cut columns
cut -d ":" -f1,3,7 /etc/passwd

# pretty table from passwd
tr ":" " " < /etc/passwd | column -t

# awk first + last fields
awk '{print $1, $NF}'

# sed find/replace (global)
sed 's/old/new/g'

# listening sockets (IPv4, non-localhost)
ss -l -4 | grep -v "127\.0\.0" | grep LISTEN | wc -l

# unique URLs from page
curl -s https://example.com | grep -Eo 'https?://[^" ]+' | sort -u

🗂️ Appendix: Why $NF Rocks in awk

• $1 → first field
• $2 → second field
• ...
• $NF → last field (works even with variable field counts)

This lets you robustly grab the “rightmost” value (e.g., shell path) without counting fields.