Security Post-it #5 – Looking for secrets
In this short security post-it, I explain how to look for secrets in files or outputs.
Oct 31, 2021 by Nicolas Béguier
During a security analysis, you often deal with outputs and look for secrets: Git repositories, logs, executable, etc.
You may have to ask yourself before what kind of secrets you are looking for, is that hidden password in code or in a log file ? A malware hidden on disk or IP inside a discovered malware.I will cover three well-known tools: Shhgit, Yara and the old-fashioned grep.
Shhgit
Shhgit is a go binary that can handle git repository, but also regular files in directories. Here it's the git repository of shhgit:
https://github.com/eth0izzle/shhgit.
This tool is looking for filenames, extensions or contents matching a specific regex: usernames and passwords, API tokens, or private keys.
How to install
Install Go for you platform. $ go get github.com/eth0izzle/shhgit # Don't forget to pick a directory for your shhgit configuration
# in this example, it's /opt/shhgit/
$ curl -o /opt/shhgit/config.yaml https://raw.githubusercontent.com/eth0izzle/shhgit/master/config.yaml
How to use
Your shhgit local path is /opt/shhgit/ and you will scan your current directory recursively $ shhgit -local . -config-path /opt/shhgit/
This tool is looking for filenames, extensions or contents matching a specific regex: usernames and passwords, API tokens, or private keys.
How to install
Install Go for you platform. $ go get github.com/eth0izzle/shhgit # Don't forget to pick a directory for your shhgit configuration
# in this example, it's /opt/shhgit/
$ curl -o /opt/shhgit/config.yaml https://raw.githubusercontent.com/eth0izzle/shhgit/master/config.yaml
How to use
Your shhgit local path is /opt/shhgit/ and you will scan your current directory recursively $ shhgit -local . -config-path /opt/shhgit/
Yara
Yara is another good way to find secrets, but focusing on the file content. Not necesseraly passwords or private keys, but hidden stuff like malware, Yara work way better than Shhgit with binaries.How to install
# On Debian/Ubuntu
$ apt install yara # On MacOS
$ brew install yara Then, you need some yara rules, I can advise these two repositories: # Default yara rules
$ git clone https://github.com/Yara-Rules/rules # Signature-Base is the YARA signature and IOC database
$ git clone https://github.com/Neo23x0/signature-base Put these repositories somewhere and keep them updated.
How to use
With a simple rule, by default yara will scan a directory or all files in it. Two major arguments can be used: -r to recursively scan the subdirectories, and -s -m to display also the matching string in the file and the rules metadata. # Run my_rule.yar in the current directory
$ yara -w -p 4 -e my_rule.yar . # Recursively
$ yara -w -r -p 4 -e my_rule.yar . # Verbose output
$ yara -w -p 4 -e -s -m my_rule.yar .
Examples
# Looking for a malware in your home directory
$ find /opt/YARA/signature-base/yara/crime_* -name "*.yar" -exec yara -w -p 4 -e {} ~ \; 2>/dev/null # Extract all URLs recursively in the current directory
$ yara -w -p 4 -r -s /opt/YARA/rules/utils/url.yar . | grep url_regex | awk '{print $2}' | sort -u
grep
Everybody know how grep works, and it's a built-in. Some well-crafted regex it can be as powerful as any other tools. However, it's probably not the fastest.Instead of grep, I will use egrep. grep is used for simple patterns and basic regular expressions (BREs); egrep can handle extended regular expressions (EREs).
How to use
Some options are useful, -H display the filename and -o extracts the matching output.
Also, two ways to use grep in general, after a pipe or with the filename in arguments. # By default, all pdf in the current directory
$ egrep -H -e 'REGEX' *.pdf # With a find
$ find . -type f -exec egrep -H -e 'REGEX' {} \; # After a pipe
$ tail -f /var/log/app.log | egrep -e 'REGEX'
Examples
Now, I can give you some useful regex. You can prefix and suffix .{5} to extract the 5 caracters before and after, it can help to understand the context and eliminate false positives.
You can try to create new one or upgrade these by using regex 101 sandbox. # IPv4
"(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)" # URL
'(http|https)://[^/"]+' # email
"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b" # MD5 hash, for SHA1 it's 40 caracters long
"[a-fA-F0-9]{32}" # secrets (very noisy)
"pass|key|api|secret" # IBAN
'[A-Z]{2}[0-9]{2}[A-Z0-9]{21}' # Base64
'[A-Za-z0-9+/]{4}*([A-Za-z0-9+/]{4}|[A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==)' # Find sqlite databases
$ find . -name '*db' -o -name '*.sql[a-z]+' $ find . -type f -exec sh -c "file \"{}\" | grep SQLite" \;