About Shell basics, Grep and Find commands

~Ganesh, Ranjith

Shell – also called as command interpreter

Interactive use: reads command lines from a terminal

Shell script: When we put command lines into a file, that file is called a shell script or shell program.

 

Broad types of shell – bash and csh

C shell (csh):

Especially good for working on a terminal

 

Bourne Shell (sh):

Probably used more often for shell programming.

Newer version “Bourne-again” shell (bash) combines handy interactive C shell−like features with Bourne shell syntax, and is preferred choice.

 

which :

It takes one or more arguments. For each of its arguments it prints to stdout the full path of the executables that would have been executed when this argument had been entered at the shell prompt. It does this by searching for an executable or script in the directories listed in the environment variable PATH

e.g.

which ls

/usr/bin/ls

 

To see which shell I am running…

echo $SHELL

/bin/bash  – tells which shell we are using (bash here).

 

env :

env is used to print environment variables.

Most environment variables are in capital

 

 

 

env

SHELL=/bin/bash

WSL_DISTRO_NAME=Ubuntu-20.04

NAME=DESKTOP-HABEOE2

PWD=/mnt/e/unix

LOGNAME=ganesh

MOTD_SHOWN=update-motd

HOME=/home/ganesh

LANG=C.UTF-8

 

HOME:

Gives our home directory

echo $HOME

/home/ganesh

 

PATH:

Tells all the directories in which binary files can be extracted

e.g.

echo $PATH

 

Find Command :

Used to find files and directories and perform subsequent operations on them. It supports searching by file, folder, name, creation date, modification date, owner and permissions

find -maxdepth 2 -iname “pledge.*” -type f

./pledge.txt

./pledge.txt.bak

-maxdepth <num> ; at most <num> searches files/directories in hierarchy.

-iname searches irrespective of the case.

-type f specifies the input type is a file.

-mindepth <num> ; at least <num> searches files/directories in hierarchy

-group <gname> ; find files/directories in which group have access permission

-user <uname> ; find files/directories in which user have access permission

-size ; find files/directoriesbased on size

-delete ; to delete the found files/directories

-atime -<min> | +<min> | <min> ; find the file which accessed in at given minutes <min> | less than given minutes (-<min>) | greater than given minutes (+<min>)

-atime ; similarly for days

-ctime, -cmin ; similarly for changed files

-mtime ; similarly for modified files


-exec ; to execute shell command on founded files/directories

find -maxdepth 2 -iname “pledge.*” -type f -exec cat {} \;

find -maxdepth 2 -iname “pledge.*” -type f | xargs cat


find -empty:

./cat

./empty_file.txt

finds all empty folders and files in the entered directory or sub-directories.

 

Grep:

grep searches for PATTERNS in each FILE.  PATTERNS is one or more patterns separated by newline characters, and grep prints each line that matches a pattern.  Typically, PATTERNS should be quoted when grep is used in a shell command.

A FILE of “-” stands for standard input.   If no FILE is given, recursive searches examine the working directory, and nonrecursive searches read standard input.

grep <Option> <SearchText/Reg Expression> <Target file/ Path > ; general form

options:

-E ; extended regular expression.

-i ; ignore case

-v ; invert

-l ; list files which has the text/pattern

-r ; recursive search on all files in given <path>

-R ; recursive dereference (open symbolic link).

-c ; count matched text/pattern

-n ; print output with line number

-e ; search for multiple pattern

Example: grep -e “-e” -e “[a-zA-Z]*nary” <target> ; search for “-e” and string end with “nary”

-B <number> ; print <number> lines before matched line

-A <number> ; print <number> lines after matched line

-C <number> ; print <number> lines before and after matched line

-f ; take each line in a file as pattern è Example : grep -f <pattern file> <Target file/ Path>

 

Business Simulation

~Shimalini and Aravindh

Business Simulation is an experimental learning tool where participants learn by running a virtual business in an interactive, risk-free and hyper – realistic environment. It helps in practising and improvising business skill such as Business insights, market analysis, operations, decision making, problem-solving, communication and leadership.

The advantages of this tool are:

  • Puts theoretical concepts into practice
  • Increases decision-making skills
  • Boosts participants engagement 
  • Provides a practical environment for soft skill development
  • Allows us to overcome loss aversion

Last week we had the opportunity to do the business simulation with the overall team. It was a different experience where we learnt a lot of things. The interaction happened through zoom meetings. We got the chance to interact with the Quilt.ai team globally. Each round of competition indicated one production year. The simulation system consisted of four sections: Research and Development, Operations, Marketing and Finance. There were four countries to set up our manufacturing plants – the United States, Germany and China. The game consisted of six years to develop their company’s productivity, capital, market, etc. 

The Capsim business tool is beneficial to both participants and instructors. The initial training and assessment gave us basic knowledge to operate the game. In the first three rounds, we were randomly tuning the values to get good results, encountered lot of difficulties, like running out of funds and triggering emergency loans. After 3rd round, we had some clarification in the process improved our performance by analysing the feedback, comparing other team’s performance and made conscious decisions. 

The overall process was very new to us. There were experienced members in my team; I had the opportunity to interact, learn and hear their thoughts. Our part in this simulation was minimum. The key takeaway from this business tool is that it allowed us to realise that we should undertake calculated risks. We are looking forward to having a similar experience in the future.

SED command

~Saranya

sed ‘s/find/replace/’ filename

Find and replace a word in a file

Ex output:

sed ‘s/find/replace/1’ filename

It will find and replace in line 1

sed ‘1,3 s/find/replace/1’ filename

It will find and replace with the given range

Ex output:

sed ‘s/find/replace/g’ filename

Find and replace all the occurrences in a line.

sed ‘1,5 s/setof/set/g’ auto.tcsh |sed ‘10,15 s/e/0000/g’  |head -15

Replace the string within the given range.

Unix commands Session

~Ranjith

In the layout team, we are learning UNIX command with the support of Sanjeev to improved our skills for automating tasks.

Session on process, kill, disk utility, alias, pushd, popd, tar, and gzip commands:

Process command:

To find process which are all running use “ps” command

ps -ef ; print each (all) process

ps -u <userName>; print process by user

ps -g <groupName>; print process process bu group

groups <username> ; to find group in which user is.

Kill Command:

Used to kill process

kill <signal> <pid> ; send a signal to process <pid>.

kill -l ; signal list;

kill -9 <pid>; -9 is a kill signal

kill -9 -1 ; all process it can.

xkill ; Select the window whose client you wish to kill with button 1….

Disk utility commands:

du -b <path> ; print file space in bytes

du -m <path> ; print file space in megabytes

du -k <path> ; print file space in kilobytes

du -h <path> ; print file space in human readable form.

du -d <depth> -h ; print all files space in given <path> with a given depth.

df -h ; print mounted disk space detail in human readable form

  • To find biggest file:
    • du -b | sort -nr ; sort file based size (bytes), In sort è n = combine digits into number, r = reverse order
  • To find smallest file:
    • du -b | sort -n ; sort file based size (bytes), In sort è n = combine digits into number
Send process into background and bring it foreground:

Press Ctrl+z to suspend process.

Type bg to send process into background

Type “jobs” to get job ID.

Type “fg %<job ID>” to bring process into foreground

Alias: set a alternate name to a command or a group of command within quotes separated “;”

alias ls ls -la –color=auto ; set “ls” as a “ls -la –color=auto”.

alias DoYouWantContinue ‘echo “\nDo you want to continue (Y/N):” ; set input = $< ; if($input != “Y” && $input != “y”) exit’ è set DoYouWantContinue as a set of command separated by “;”.

Unalias <alias Name> ; to remove alias;

/<command> ; to run raw command; even when command name is aliased to something else.

Change directory :

cd <path> ; change directory to given path.

cd – ; last cd path.

pushd <path> ; push directory into stack and cd to it.

pushd +<rotate stack number> ; rotate the stack

popd ; pop the top directory in stack

Create Archieve:

tar -cf <archive.tar> <directory> ; Create archive.tar from <directory>, it is like git version control.

tar -tvf <archive.tar> ; List all files in archive.tar verbosely.

tar -xf <archive.tar> ; Extract all files from archive.tar.

tar -d -f <archieve.tar> <directory>; show the difference between tar and files in directory.

tar -u -f <archieve.tar> <directory>; create a new entry for all files with new time stamp(if file is modified).

tar -r -f <archieve.tar> <directory>; append a new file in tar file.

tar -xvf auto.tar test.tcsh –occurrence=2 ; extract 2nd occurrence of test.tcsh file in auto.tar tree.

Compress:

gzip -c <file> ; compress file

gzip -d <file> ; decompress file

gunzip <file> ; unzip file

 

US Election Model

~Sharat Kumar & Vasanth

The power of predictive modelling and Machine Learning in particular. We had an opportunity to put it to test, by working on the prediction of the US election, a couple of weeks prior to the US Presidential Election. WE approached this using supervised learning. First, we considered two states of America, Texas (Republican/healthy Red state) and California (Democratic/ strong Blue state). We scraped around 10000 tweets for 5 Days from twitter for both the states. We tagged red as Pro-Trump/Anti-Biden and treated Blue as  Pro-Biden/Anti-Trump. 

We selected 100 top Fans of Biden and Trump across the US, and we scraped tweets relevant to US elections from those handles for one day. We trained a model using those tagged data into an AI machine. After training, the tool considers 70% of the data to train and the rest 30% for validation and testing. During the process, a confusion matrix is formed. A confusion matrix is a table describing the performance of a classification model (or “classifier”) on a set of test data for which the correct values are known. 

This table shows how often the model classified each label correctly (in blue), and which labels were most often confused for that label (in grey). 

We created two models out of them, in one model, when we feed an individual’s tweet history into the machine, it will identify whether the person is a Trump/Biden fan. Another model is when we provide the Twitter data of a particular state; the model predicts who has more support in that state. We used the second model, and we fed the twitter data of 11 swing states in the US. 

States Red Blue Predicted Result Actual Result
Concord (New Hampshire) 36.48% 63.52% Blue Blue
Florida 47.29% 52.71% Blue Red
Iowa 51.01% 48.99% Red Red
Michigan 31.34% 68.66% Blue Blue
Minnesota 43.79% 56.21% Blue Blue
Nevada 37.22% 62.78% Blue Blue
Ohio 42.32% 57.68% Blue Red
Pennsylvania 43.18% 56.82% Blue Blue
Raleigh (North Carolina) 46.27% 53.73% Blue Red
Virginia 44.80% 55.20% Blue Blue
Wisconsin 43.01% 56.99% Blue Blue

In 8 states among the 11, we predicted the right result. We got an accuracy of 72.72% as our result. 

This gave a first hand experience of the power of AI and Machine learning. While these tools can be used to predict and prescribe inputs for various business decisions it can also be an effective platform for addressing several complex societal problems. We are excited to engage and learn more on the underlying capabilities these new age platforms offer.