Linux Stuff |
|
I am no linux guru but having to work with Postgres on Linux and now Hadoop I thought it would be best if kept of list of stuff I am learning.
Basic Commands
Here are a few basic commands I find myself using a lot..
du -h
Finding the size of a directory. -h means human readable format
df - h
Finding the disk free space / disk usage
CTRL-D to close terminal
PWD returns present working directory or print working directory
Tilda (~) means you are at the home directory
CTRL-L to clear the screen
-F returns file type
ls -lrt list of files in reverse order and ordered by timestamp
Wilcards
du -h
Finding the size of a directory. -h means human readable format
df - h
Finding the disk free space / disk usage
CTRL-D to close terminal
PWD returns present working directory or print working directory
Tilda (~) means you are at the home directory
CTRL-L to clear the screen
-F returns file type
ls -lrt list of files in reverse order and ordered by timestamp
Wilcards
- ls -l /dev/sda* quick way to see all associated directories
- ls -l /dev/sda? returns all directories with an additional single character after "sda"
- ls -l /dev/sda[12] returns "sda1" or "sda2"
Remove more than one file at a time with Grep or Find |
|
I needed to remove a list of files from the pg_xlog directory. Asked a colleague for some help on this.
dir | grep "your filter" | awk '{print $9}' | xargs rm -f
So this was explained to me as.
use grep to filter. awk returns just the column listing you want to remove.
xargs takes what you have in the pipe and then runs rm -f
-f will force the removal with no prompt.
you can also use gawk as well instead of awk. not really sure of the difference.
I like this better. So this looks in the current directory for files with the extension .tar.
It also looks for the modification time between Apr 5th to April 11th and then deletes those files.
find . -type f -name "*.tar" -newermt "Apr 5" ! -newermt "Apr 11" -delete
Find will recurse through the directory. include the maxdepth parameter to only go one level down
find /etc -maxdepth 1 -type l
Use the find command to find files larger than 20MB and execute du -h for each file found
find /<directory> -size +20000k -type f -exec du -h {} \;
I need to find the last backup file that was taken of a Postgres database. Since I can't find a way to do that inside of Postgres I am left using linux command line tools.
this will find the last backupfile
find <directory> -type f -name "*tar.gz" | sort | tail -1
Find the backs that are 7 days or older. -ctime is creation time
find -type f -name "name of file" -ctime +7
dir | grep "your filter" | awk '{print $9}' | xargs rm -f
So this was explained to me as.
use grep to filter. awk returns just the column listing you want to remove.
xargs takes what you have in the pipe and then runs rm -f
-f will force the removal with no prompt.
you can also use gawk as well instead of awk. not really sure of the difference.
I like this better. So this looks in the current directory for files with the extension .tar.
It also looks for the modification time between Apr 5th to April 11th and then deletes those files.
find . -type f -name "*.tar" -newermt "Apr 5" ! -newermt "Apr 11" -delete
Find will recurse through the directory. include the maxdepth parameter to only go one level down
find /etc -maxdepth 1 -type l
Use the find command to find files larger than 20MB and execute du -h for each file found
find /<directory> -size +20000k -type f -exec du -h {} \;
I need to find the last backup file that was taken of a Postgres database. Since I can't find a way to do that inside of Postgres I am left using linux command line tools.
this will find the last backupfile
find <directory> -type f -name "*tar.gz" | sort | tail -1
Find the backs that are 7 days or older. -ctime is creation time
find -type f -name "name of file" -ctime +7
Word Count |
|
count number of lines in a file
wc -l /name of file/etc/services
wc -l /name of file/etc/services
Last Argument |
|
use the last argument
page through the file with less and use !$ to use the last argument
less !$
page through the file with less and use !$ to use the last argument
less !$
exit looking through a file |
|
when looking through a file and you need to exit.
q
q
Looks through the top 10 lines of a file |
|
head /etc/services
to view the top 3 lines add -n
head -n 3 /etc/services
to view the top 3 lines add -n
head -n 3 /etc/services
Look through the bottom 10 lines of a file |
|
tail /etc/services
to view bottom 3 lines add -n
tail -n 3 /etc/services
to view bottom 3 lines add -n
tail -n 3 /etc/services
Filter results with GREP |
|
the carrot filters for the beginning of a string
the following command filters for strings starting with kernel
yum list installed | grep ^kernel y
Search for a word boundary in the ntp.conf file meaning looking for the specific word server
grep '\bserver\b' ntp.conf
Search for words at the end of a string via the $
grep -E 'ion$' /usr/share/dict/words
Search for words beginning with a certain string with two unknowns and ends with a certain string
^ searches the beginning of a string
$ searches the end of a string
combine ^$ to find empty lines
grep -E '^po..ulte$' /usr/share/dict/words
Search for words with 3 vowels in a row
grep -E '[aeiou]{3}' /usr/share/dict/words
the following command filters for strings starting with kernel
yum list installed | grep ^kernel y
Search for a word boundary in the ntp.conf file meaning looking for the specific word server
grep '\bserver\b' ntp.conf
Search for words at the end of a string via the $
grep -E 'ion$' /usr/share/dict/words
Search for words beginning with a certain string with two unknowns and ends with a certain string
^ searches the beginning of a string
$ searches the end of a string
combine ^$ to find empty lines
grep -E '^po..ulte$' /usr/share/dict/words
Search for words with 3 vowels in a row
grep -E '[aeiou]{3}' /usr/share/dict/words
Install updates to Linux |
|
install update to your linux OS. Must run as root
yum install updates
yum install updates
Copy a File to current directory |
|
copy a file to the current directory
cp <filename> .
cp <filename> .
Exit VI |
|
I still think VI sucks but anyway.
need to exit VI
its colon then q!
so
:
q!
need to exit VI
its colon then q!
so
:
q!
Split a string in Linux |
|
need to split a line with this string to return just user name
-f2 returns the second element or the element after the split
pguser = 'user name '
grep pguser | cut -d '=' -f2
-f2 returns the second element or the element after the split
pguser = 'user name '
grep pguser | cut -d '=' -f2
Copy Files from local machine to remote machine |
|
I need to move files from a local machine, in this case a Mac Book Pro to a Hadoop Cluster Linux server..
scp <local path to file including filename> <username on remote server@Fully qualified domain name:/file destination>
Here is an example to make it easier to understand
scp /Users/mdspain/desktop/somefile.txt michaeldspain@FQDN:/mnt/home/michaeldspain/desktop
scp <local path to file including filename> <username on remote server@Fully qualified domain name:/file destination>
Here is an example to make it easier to understand
scp /Users/mdspain/desktop/somefile.txt michaeldspain@FQDN:/mnt/home/michaeldspain/desktop
For Loop in Bash |
|
Trying to learn some Bash. Seems like same concepts as powershell.
In my trying to learn it, I wrote a loop to convert a bunch .avi files to .mp4
Here is the syntax as executed from the directory that the files are located. ffmeg is a command line utility to convert video files.
for i in *.avi
do
ffmpeg -i "$i" "${i%.avi}.mp4"
done
In my trying to learn it, I wrote a loop to convert a bunch .avi files to .mp4
Here is the syntax as executed from the directory that the files are located. ffmeg is a command line utility to convert video files.
for i in *.avi
do
ffmpeg -i "$i" "${i%.avi}.mp4"
done