- redirect
- time & date
- download and extract
- compress
- show 256 colors
- commands
- batch commands
- ldapsearch
- others
redirect
show stdout but redirect all to file
[!NOTE|label:references:]
- redirect overview
|| visible in terminal || visible in file || existing Syntax || StdOut | StdErr || StdOut | StdErr || file ==========++==========+==========++==========+==========++=========== > || no | yes || yes | no || overwrite >> || no | yes || yes | no || append || | || | || 2> || yes | no || no | yes || overwrite 2>> || yes | no || no | yes || append || | || | || &> || no | no || yes | yes || overwrite &>> || no | no || yes | yes || append || | || | || | tee || yes | yes || yes | no || overwrite | tee -a || yes | yes || yes | no || append || | || | || n.e. (*) || yes | yes || no | yes || overwrite n.e. (*) || yes | yes || no | yes || append || | || | || |& tee || yes | yes || yes | yes || overwrite |& tee -a || yes | yes || yes | yes || append- Redirect terminal output to file
- Bash: Redirect stdout and stderr
- show Command Output Redirection.pdf
- 3.6 Redirections
- * Beyond Linux From Scratch
tips:
- echo to stderr
$ echo 'abcdefg' >/dev/null $ echo 'abcdefg' >/dev/null >&2 abcdefg
$ bash -c "echo a;bahs;echo b;bhas" >>file 2> >( tee -a file >&2 )
bash: line 1: bahs: command not found
bash: line 1: bhas: command not found
$ cat file
a
bash: line 1: bahs: command not found
b
bash: line 1: bhas: command not found
stderr output with filter
$ bash -c "echo a;bahs;echo b;bhas" >>file 2> >( tee -a file 2>&1 | grep -v bahs >&2 ) bash: line 1: bhas: command not found # or $ rm -rf file; bash -c "echo a;bahs;echo b;bhas" >>file 2> >( tee -a file | grep -v bahs >&2 ) bash: line 1: bhas: command not found $ cat file a bash: line 1: bahs: command not found b bash: line 1: bhas: command not foundor
$ bash -c "set -e; echo a;bahs;echo b;bhas" >>cmd.out 2> >( tee -a cmd.out >&2 ) bash: line 1: bahs: command not found $ cat cmd.out a bash: line 1: bahs: command not found
time & date
[!TIP|label:see also:]
show date with timezone
# with quotes
$ TZ=':Asia/Shanghai' date
# or without quotes
$ TZ=America/Los_Angeles date
show cal
$ cal -y
# or
$ cal -y |
tr '\n' '|' |
sed "s/^/ /;s/$/ /;s/ $(date +%e) / $(date +%e | sed 's/./#/g') /$(date +%m | sed s/^0//)" |
tr '|' '\n'
2014
January February March
Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa
1 2 3 4 1 1
5 6 7 8 9 10 11 2 3 4 5 6 7 8 2 3 4 5 6 7 8
12 13 14 15 16 17 18 9 10 11 12 13 14 15 9 10 11 12 13 14 15
19 20 21 22 23 24 25 16 17 18 19 20 21 22 16 17 ## 19 20 21 22
26 27 28 29 30 31 23 24 25 26 27 28 23 24 25 26 27 28 29
30 31
April May June
Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa
1 2 3 4 5 1 2 3 1 2 3 4 5 6 7
6 7 8 9 10 11 12 4 5 6 7 8 9 10 8 9 10 11 12 13 14
13 14 15 16 17 18 19 11 12 13 14 15 16 17 15 16 17 18 19 20 21
20 21 22 23 24 25 26 18 19 20 21 22 23 24 22 23 24 25 26 27 28
27 28 29 30 25 26 27 28 29 30 31 29 30
July August September
Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa
1 2 3 4 5 1 2 1 2 3 4 5 6
6 7 8 9 10 11 12 3 4 5 6 7 8 9 7 8 9 10 11 12 13
13 14 15 16 17 18 19 10 11 12 13 14 15 16 14 15 16 17 18 19 20
20 21 22 23 24 25 26 17 18 19 20 21 22 23 21 22 23 24 25 26 27
27 28 29 30 31 24 25 26 27 28 29 30 28 29 30
31
October November December
Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa
1 2 3 4 1 1 2 3 4 5 6
5 6 7 8 9 10 11 2 3 4 5 6 7 8 7 8 9 10 11 12 13
12 13 14 15 16 17 18 9 10 11 12 13 14 15 14 15 16 17 18 19 20
19 20 21 22 23 24 25 16 17 18 19 20 21 22 21 22 23 24 25 26 27
26 27 28 29 30 31 23 24 25 26 27 28 29 28 29 30 31
30
synchronize date and time with over ssh
[!NOTE|label:inspired from:]
$ date --set="$(ssh [username]@[sshserver] date)"
- verify
$ date; ssh username@sshserver date; curl -fsSL "http://worldtimeapi.org/api/timezone/America/Los_Angeles" | jq -r .datetime Tue Apr 2 20:03:32 PDT 2024 Tue Apr 2 20:03:33 PDT 2024 2024-04-02T20:03:33.405227-07:00
download and extract
[!NOTE|label:references:]
*.gz$ wget -O - http://example.com/a.gz | tar xz*.zip$ curl -fsSL https://services.gradle.org/distributions/gradle-4.7-all.zip | bsdtar xzf - -C <EXTRACT_PATH> # with password $ curl -fsSL \ -u<user>:<passwd> \ https://path/to/file.zip | bsdtar -xzf- --passphrase <PASSWD_OF_ZIP> - -C <EXTRACT_PATH>*.tar.gz$ curl -fsSL https://path/to/file.tar.gz | tar xzf - -C <EXTRACT_PATH> # example $ curl -fsSL -j -k \ -H "Cookie: oraclelicense=accept-securebackup-cookie" \ http://download.oracle.com/otn-pub/java/jdk/8u181-b13/96a7b8442fe848ef90c96a2fad6ed6d1/jdk-8u181-linux-x64.tar.gz | tar xzf - -C '/opt/java'
check file without extract
$ tar -Oxvf myfile.tgz path/to/my.sh | less
extract jar
$ unzip <jar-name>.jar -d <target-folder>
- Delete files from JAR without unzip
$ zip -d <jar-name>.jar <path/to/file.txt>
recursive download
[!TIP] references:
params shortcuts :
SHORT PARAMS LONG PARAMS -r--recursive-m--mirror-l--level-k--convert-links-K--backup-converted-P--directory-prefix-nv--no-verbose-nc--no-clobber-nd--no-directories-nH--no-host-directories-np--no-parent-x--force-directories-b--background-v--verbose-p--page-requisites
$ wget --recursive \ # -r
--user=admin \
--password=admin \ # --ask-password
--auth-no-challenge \ # optional
--no-host-directories \ # -nH
--no-parent \ # -np
--reject '*.html*' \
--directory-prefix=./ \ # -P
--include-directories=local-dir \ # -I
http://example.com/remote-dir
# or
$ wget --recursive \ # -r
--no-parent \ # -np : will not crawl links in folders above the base of the URL
--convert-links \ # -k : convert links with the domain name to relative and uncrawled to absolute
--random-wait --wait 3 --no-http-keep-alive \ # do not get banned
--no-host-directories \ # -nH : do not create folders with the domain name
--execute robots=off --user-agent=Mozilla/5.0 \ # I AM A HUMAN!!!
--level=inf --accept '*' \ # -l : do not limit to 5 levels or common file formats
--reject="index.html*" \ # use this option if you need an exact mirror
--cut-dirs=0 \ # replace 0 with the number of folders in the path, 0 for the whole domain
$URL
- mirror whole website
$ wget -m https://www.baeldung.com/
download directly
$ wget -r -np -nH --cut-dirs=1 https://www.baeldung.com/linux
# or
$ wget -r --no-parent --no-host-directories --cut-dirs=1 https://www.baeldung.com/linux
with level
$ wget -r -np -l 2 https://www.baeldung.com/linux/ # or $ wget -r --no-parent --level=2 https://www.baeldung.com/linux/with credentials
.wgetrc$ cat ~/.wgetrc user=admin password=adminfrom cmd
$ wget --user=admin --password=admin # or $ wget --user=admin --ask-password Password for user 'admin': admin- ignore credentials
$ wget --no-check-certificate
converting links for local viewing
$ wget -r --no-parent --convert-links https://www.baeldung.com/linux/category/webswitching off robot exclusion
$ wget -r --level=1 --no-parent --convert-links -e robots=off -U="Mozilla"
compress
zip package with dot-file
.[^.]*$ zip name.zip * .[^.]*'shopt -s dotglob[!NOTE|label:references:]
$ shopt -s dotglob $ zip name.zip *.$ zip -r name.zip .
remove dot-file without skipping '..' '.' issue
[!NOTE|label:references:]
- imarslo : bash/bash
- turn
dotgloboff$ shopt -u dotglob
-
$ shopt -s dotglob $ rm [-rf] * .[^.]*$ rm [-rf] .[^.]*-
$ rm [-rf] .[!.]* -
$ rm [-rf] .??* copy all including hidden files
$ cp -r * .??* /dest
show 256 colors
[!TIP] see also:
$ for i in {0..255}; do echo -e "\e[38;05;${i}m${i}"; done | column -c 80 -s ' '; echo -e "\e[m"
# or
$ yes "$(seq 1 255)" | while read i; do printf "\x1b[48;5;${i}m\n"; sleep .01; done
# or: https://www.commandlinefu.com/commands/view/5879/show-numerical-values-for-each-of-the-256-colors-in-bash
$ for code in {0..255}; do echo -e "\e[38;05;${code}m $code: Test"; done)
# or: https://www.commandlinefu.com/commands/view/6138/show-numerical-values-for-each-of-the-256-colors-in-bash
$ for i in {0..255}; do echo -e "\e[38;05;${i}m${i}"; done | column -c 80 -s ' '; echo -e "\e[m"
# or: https://www.commandlinefu.com/commands/view/11759/show-numerical-values-for-each-of-the-256-colors-in-bash-for-bold-and-normal-fonts
$ for code in $(seq -w 0 255); do for attr in 0 1; do printf "%s-%03s %bTest%b\n" "${attr}" "${code}" "\e[${attr};38;05;${code}m" "\e[m"; done; done | column -c $((COLUMNS*2))
# better vesion
$ for code in $(seq -w 0 255); do for attr in 0 1; do printf "%b%s-%03s%b\n" "\e[${attr};38;05;${code}m" "${attr}" "${code}" "\e[m"; done; done | column -c $((COLUMNS*3))
# .. zsh ..
# or: https://www.commandlinefu.com/commands/view/5876/show-numerical-values-for-each-of-the-256-colors-in-zsh
$ for code in {000..255}; do print -P -- "$code: %F{$code}Test%f"; done
# or: https://www.commandlinefu.com/commands/view/12471/show-numerical-values-for-each-of-the-256-colors-in-zsh
for i in {0..255}; do echo -e "\e[38;05;${i}m\\\e[38;05;${i}m"; done | column -c 80 -s ' '; echo -e "\e[m"
commands
ls
[!TIP|label:references]
list numeric names
[!NOTE|label:references:]
$ ls git-[[:digit:]]*.png git-0.png git-1.png git-2.png git-3.png git-4.png git-5.png $ ls git-+([0-9])*.png git-0.png git-1.png git-2.png git-3.png git-4.png git-5.png
PWD's secrets
$ l | grep bc
lrwxrwxrwx 1 marslo marslo 37 Mar 4 00:25 bc -> /home/marslo/Tools/Git/BrowserConfig//
$ cd bc/
$ pwd -L
/home/marslo/bc
$ pwd -P
/home/marslo/Tools/Git/BrowserConfig
list the command startsWith
$ compgen -c "system-config-"
system-config-authentication
system-config-authentication
system-config-date
system-config-firewall
system-config-firewall-tui
system-config-kdump
system-config-keyboard
system-config-keyboard
system-config-network
system-config-network
system-config-network-cmd
system-config-network-cmd
system-config-network-tui
system-config-printer
system-config-printer-applet
system-config-services
system-config-services
system-config-users
fuzzy find for commands
$ apropos editor | head
Git::SVN::Editor (3pm) - commit driver for "git svn set-tree" and dcommit
INIFILE (1) - OpenLink Virtuoso Opensource ini File Editor
atobm (1) - bitmap editor and converter utilities for the X Window System
bitmap (1) - bitmap editor and converter utilities for the X Window System
bmtoa (1) - bitmap editor and converter utilities for the X Window System
ed (1) - line-oriented text editor
editor (1) - Nano's ANOther editor, an enhanced free Pico clone
editres (1) - a dynamic resource editor for X Toolkit applications
ex (1) - Vi IMproved, a programmers text editor
gedit (1) - text editor for the GNOME Desktop
batch commands
batch rename
$ l
total 4.0K
-rw-r--r-- 1 marslo marslo 10 Feb 21 00:43 a.b
$ rename -v 's/\./_/g' *
a.b renamed as a_b
$ l
total 4.0K
-rw-r--r-- 1 marslo marslo 10 Feb 21 00:43 a_b
-
$ for file in sw.ras.*; do mv "$file" "${file/ras./}"; done /usr/local/bin/renamein OSX$ /usr/local/bin/rename -v 's/_xyz.com//g' *.txt 'a_xyz.com.txt' renamed to 'a.txt' 'b_xyz.com.txt' renamed to 'b.txt' 'c_xyz.com.txt' renamed to 'c.txt' 'd_xyz.com.txt' renamed to 'd.txt' 'e_xyz.com.txt' renamed to 'e.txt' 'f_xyz.com.txt' renamed to 'f.txt' 'g_xyz.com.txt' renamed to 'g.txt' 'h_xyz.com.txt' renamed to 'h.txt'-
[!NOTE|label:references:]
$ /usr/local/opt/util-linux/bin/rename -v '_xyz.com' '' *.ttf `a_xyz.com.ttf' -> `a.ttf' `b_xyz.com.ttf' -> `b.ttf' `c_xyz.com.ttf' -> `c.ttf' `d_xyz.com.ttf' -> `d.ttf' `e_xyz.com.ttf' -> `e.ttf' `f_xyz.com.ttf' -> `f.ttf' `g_xyz.com.ttf' -> `g.ttf' `h_xyz.com.ttf' -> `h.ttf'
xargs rename
$ shopt -s extglob
$ ls git-+([0-9])*.png
git-0.png git-1.png git-2.png git-3.png git-4.png git-5.png
$ ls --color=none git-+([0-9])*.png | xargs rename -v 's/git-/git-for-windows-/'
'git-0.png' renamed to 'git-for-windows-0.png'
'git-1.png' renamed to 'git-for-windows-1.png'
'git-2.png' renamed to 'git-for-windows-2.png'
'git-3.png' renamed to 'git-for-windows-3.png'
'git-4.png' renamed to 'git-for-windows-4.png'
'git-5.png' renamed to 'git-for-windows-5.png'
batch move
[!NOTE]
-I replace-str
$ mkdir backup-folder && ls | grep -Ze ".*rar" | xargs -d '\n' -I {} mv {} backup-folder
batch copy
reference:
$ ls -1 a/b/* 11 12 | xargs cp -t copy-target-folder/
copy single file to multipule folders
$ echo dir1 dir2 dir3 | xargs -n 1 cp file1
# or
$ echo dir{1..10} | xargs -n 1 cp file1
ldapsearch
[!NOTE] enhaanced script
- marslo/mytools reference:
- ldapsearch Examples
- The ldapsearch Tool
- Querying AD with ldapsearch
-LLL # just a particular way to display the results -H ldap://wspace.mydomain.com # the URL where the LDAP server listens -x # use simple authentication, not SASL -D 'user1' # the account to use to authenticate to LDAP -w 'user1password' # the password that goes with the account on the previous line -E pr=1000/noprompt # ask the server for all pages, don't stop after one -b 'ou=mydomain,dc=wspace,dc=mydomain,dc=com' # the base of the search. We don't want results from e.g. 'ou=blah,dc=wspace,dc=mydomain,dc=com' '(&(objectClass=person)(uidNumber=*))' # Ask for any entry that has attributes objectClass=person and uidNumber has a value SAMAccountName uid uidNumber # Show only these attributes
search specific user
[!NOTE] info :
- ldap url :
ldaps://ldap.mydomain.com:636- base search base :
dc=mydomain,dc=com- login user :
user1/user1password- search :
user2
- remove
#.*$ ldapsearch ... | sed -r '/^(#.*)$/d'- remove empty lines
$ ldapsearch ... | sed -r '/^\s*$/d'remove all
$ ldapsearch ... | sed -r '/^(#.*)$/d;/^\s*$/d' # or $ ldapsearch ... | sed -r '/(^#.*)|(^\s*)$/d'
$ ldapsearch \
-LLL \
-x \
-H 'ldaps://ldap.mydomain.com:636' \
-b 'dc=mydomain,dc=com' \
-D 'user1' \
-w 'user1password' \
CN='user2'
- or insert password via interactive mode (
-W)$ ldapsearch \ -LLL \ -x \ -H 'ldaps://ldap.mydomain.com:636' \ -b 'dc=mydomain,dc=com' \ -W \ -D 'user1' \ CN='user2'
filter DN field only
$ ldapsearch \
[-LLL \]
-H 'ldaps://ldap.mydomain.com:636' \
-b 'dc=mydomain,dc=com' \
-x \
-D 'user1' \
-w 'user1password' \
CN='user2' \
DN
filter SAMAccountName, uid and uidNumber only
[!TIP] filter base on base DN (
OU=Person,DC=mydomain,DC=com)
$ ldapsearch \
-LLL \
-x \
-H 'ldaps://ldap.mydomain.com:636' \
-b 'ou=Workers,dc=mydomain,dc=com' \
-D 'user1' \
-w 'user1password' \
-E 'pr=1000/noprompt' \
'(&(objectClass=user)(sAMAccountName=*))' \
SAMAccountName uid uidNumber DN
filter particular group
$ ldapsearch \
-x \
-H 'ldaps://ldap.mydomain.com:636' \
-b 'OU=DL,OU=Groups,OU=GLOBAL,OU=Sites,dc=mydomain,dc=com' \
-D 'user1' \
-w 'user1password' \
-E 'pr=1000/noprompt' \
'(&(objectClass=group)(CN=*))'
- search particular group (
cn=DL-name-group)$ ldapsearch \ -x \ -H 'ldaps://ldap.mydomain.com:636' \ -b 'OU=DL,OU=Groups,OU=GLOBAL,OU=Sites,dc=mydomain,dc=com' \ -D 'user1' \ -w 'user1password' \ -E 'pr=1000/noprompt' \ CN='DL-name-group'
get userCertificates
[!NOTE|label:references:]
- Problems with ldap userCertificate attribute
# convert a pem certificate into der openssl x509 -outform DER -in incert.pem -out outcert.der # created LDIF file ldif -b "usercertificate;binary" < outcert.der > cert.ldif # creates an usercertificate attribute encoded in base64 ldapmodify -x -W -D "cn=Manager,dc=yourorg,dc=com" -f cert.ldif
get cert info
$ ldapsearch marslo userCertificate | awk '{print $NF}' | xargs -i bash -c "echo {} | base64 -d -w0 | openssl x509 -noout -dates -subject -issuer" # or $ while read -r crt; do echo "${crt}" | base64 -d -w0 | openssl x509 -noout -dates -subject -issuer; done < <(ldapsearch marslo userCertificate | awk '{print $NF}') notBefore=Apr 28 18:05:13 2023 GMT notAfter=Apr 27 18:05:13 2025 GMT subject=DC = com, DC = example, OU = Workers, CN = marslo, emailAddress = marslo@example.com issuer=DC = com, DC = example, CN = example SC Issuing CA V1save local
- der
$ while read -r n c; do echo "-- ${n} --"; echo "${c}" | base64 -d -w0 > cert_${n}.der; done < <(ldapsearch marslo userCertificate | awk '{print $NF}' | cat -n) - crt
$ while read -r n c; do echo "-- ${n} --"; echo "${c}" | base64 -d -w0 > cert_${n}.der; openssl x509 -in cert_${n}.der -inform DER -out cert_${n}.crt; done < <(ldapsearch marslo userCertificate | awk '{print $NF}' | cat -n)
- der
others
directory diff
$ diff --suppress-common-lines -y <(cd path_to_dir1; find .|sort) <(cd path_to_dir2; find .|sort)
show some command periodically
$ watch --interval 1 ls -alt
- watch with pipe
$ watch -n 1 'ls -Altrh | grep <keywords>'
clear
$ printf "\ec"
use less as tail -f
$ less +F <filename>
netcat & nmap-ncat
[!NOTE] references:
install
$ sudo yum -y install epel-release [yum-utils] # or via url $ sudo dnf [re]install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm $ sudo dnf [re]install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm $ sudo yum update -y $ yum list | grep netcat netcat.x86_64 1.219-2.el8 @epel $ sudo yum install -y netcat.x86_64 # check $ sudo yum repolistswitch with nmap & netcat
$ ls -altrh /usr/local/bin/nc lrwxrwxrwx 1 root root 22 Mar 14 03:14 /usr/local/bin/nc -> /etc/alternatives/nmap $ sudo update-alternatives --config nmap There are 2 programs which provide 'nmap'. Selection Command ----------------------------------------------- *+ 1 /usr/bin/ncat 2 /usr/bin/netcat Enter to keep the current selection[+], or type selection number: 2check
# by using netcat $ nc -zv google.com 443 Connection to google.com (142.251.214.142) 443 port [tcp/https] succeeded! # by using nact $ ncat -zv google.com 443 Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat: Connected to 142.251.214.142:443. Ncat: 0 bytes sent, 0 bytes received in 0.07 seconds.check package
$ rpm -ql netcat.x86_64 /usr/bin/nc /usr/bin/netcat /usr/lib/.build-id /usr/lib/.build-id/f3 /usr/lib/.build-id/f3/3de6290429f99a8d8f5fe646a93bcc952dafdd /usr/share/man/man1/nc.1.gz /usr/share/man/man1/netcat.1.gz $ rpm -ql nmap-ncat.x86_64 /usr/bin/nc /usr/bin/ncat ...
alternatives & update-alternatives
install
$ sudo update-alternatives --install /usr/bin/java java /opt/java/jdk1.8.0_121/bin/java 999 $ sudo update-alternatives --auto java $ sudo update-alternatives --install /usr/bin/javac javac /opt/java/jdk1.8.0_121/bin/javac 999 $ sudo update-alternatives --auto javacmodify
$ ls -altrh $(which -a nc) lrwxrwxrwx 1 root root 22 Jun 1 04:02 /usr/bin/nc -> /etc/alternatives/nmap $ sudo alternatives --config nmap There are 2 programs which provide 'nmap'. Selection Command ----------------------------------------------- *+ 1 /usr/bin/netcat 2 /usr/bin/ncat Enter to keep the current selection[+], or type selection number: 1