Lecture 2 - Command line and Stack¶

from IPython.display import IFrame
from IPython.display import Markdown
# Additional styling ; should be moved into helpers
from IPython.core.display import display, HTML
HTML('<style>{}</style>'.format(open('styler.css').read()))

Announcements!¶

Starting Wednesday, we will use Collaborate Ultra for labs (it just makes most sense)

Change to the lab schedule (Wednesday and Thursday)
- 8:30AM-9:30AM - Lab session 1 (virtual only) on Collaborate Ultra
- 9:30-12:30 - Class on Zoom (link in Canvas)
- 12:30-1:30 - Lunch Break
- 1:30 - 3:30 Lab session 2 (physical and virtual) on Collaborate Ultra

We will use the Queue system for our labs (for both online and physics labs!)

I have created a Collaborate Ultra room for all of you to use at any time (it will also be our lab room) - you are all moderators in the session
- Here is the link: https://ca.bbcollab.com/guest/e4d39e4dfc3648bfbc1a26fe9c773623

Lecture Outline¶

Overview of the Platforms used in this program (15 mins)

The Command Line - Part 1 (25 mins)

Break (10 mins)

The Command Line - Part 2 (15 mins)

Walk-through submitting a lab on GitHub Classroom and Canvas (Will post video)

1. Overview of the Platforms used in this program¶

Tool	Purpose
UBC Student Email	We need this for Office365; good for jobs
Web browser	Recommend Chrome; Realistically good to have multiple browsers
LastPass password manager	Maintain sanity with passwords
Slack	You all know what this is good for
Visual Studio Code	Code editor for 530, 531, 540 + viz courses
VS Code extensions	GitLens (git stuff); Linters, syntax checkers
GitHub	This is where all your code is stored; all your project courses will be here
Git	Version control system

Tool	Purpose
Python	1 of 2 programming languages
Essential python packages	numpy, pandas, seaborn, sci-kit-learn, others…
Conda	Environment manager (base)
JupyterLab and Jupyter Notebooks	Platform for running notebooks
R; XQuartz; IRkernel	Second of 2 languages used in the program
RStudio	IDE for R
LaTeX	Typesetting language for Math/Sciences
PostgreSQL	Databases
Docker	Collaborative Software Dev/Workflows
Post-installation notes	Script for checking if everything is installed correctly

The Command Line - Part 1¶

What is the Command Line?¶

The command line is the text interface to the computer that accepts commands that the computer will execute. These commands include:

starting programs
navigating directories and manipulating files
searching, sorting, and editing text files
system and environment configuration

Why learn command line?¶

The command line is the text interface to the computer. Understanding the command line allows you to interact with the computer in ways that you often cannot with the graphical user interface (GUI).

The command line is commonly used for scripting and automation of tasks and when accessing remote systems. It will also be useful to run programs that make use of the command line (eg. github).

Why use command line?¶

The command line is part of the operating system (OS), which is software that manages your computer including all devices and programs.

Common operating systems include Windows, Mac OS, and Linux/Unix.
Some commands will be OS specific

You might be wondering why we would ever prefer command line over using the graphical user interface (GUI).

Certain tools may only be available to command line.
Sometimes command line is faster.

Command Line on Windows¶

The command line on Windows dates back to the original Microsoft operating system called DOS (Disk Operating System) in 1981.

This command line interface is still part of all modern Windows operating systems and is accessible as the “Command Prompt”.

I am recommending you use GitBash instead of the default Windows Command Prompt.

display(Markdown('#### Sli.do poll'))
IFrame('https://app.sli.do/event/0oandysw/embed/polls/a249631b-a9de-4bf0-acaf-7211002d48db',
       width=800, height=400)

System Message: WARNING/2 (/home/runner/work/data530_course/data530_course/class/Lecture2.ipynb, line 1)

Non-consecutive header level increase; 0 to 4

Windows Terminal¶

This is what the GitBash “Terminal” looks like. From here on out, whenever we say “Terminal”, we mean “GitBash on Windows”, “Terminal on macOS and Ubuntu”

The directory separator is a forward slash / for Macs/Ubuntu, and the opposite pointing slash on Windows \
However if you use GitBash, this should preserve your sanity and you should be able to use forward slash on Windows as well
In fact, many of the macOS/Ubuntu commands will also work in GitBash, but for completeness we will also share the other commands

Image source: Git for windows

Mac OS Command Line¶

The command line for Mac OS uses the same commands as Linux. It can be opened using Finder then Utilities then Terminal.

Alternatively, we could type Cmd + spacebar, then type “Terminal” and press ENTER

Entering Commands¶

Enter a command at a prompt. The prompt may be a > or a $ or customized by the user.

Press ENTER to execute the command.

On Windows, commands are mostly case-insensitive while on Mac/Linux they are case-sensitive.

File System¶

The file system organizes data on a device as a hierarchy of directories and files (like a tree). Each folder (AKA directory) has a name and can contain any number of files or subdirectories. Each file has a name. The user can change (navigate) directories in the hierarchy.

File System¶

The tree is rooted at, well, the root.

There is only one root of a directory hierarchy. Every item in the tree is either a file or a directory (AKA folder).
You can think of a directory as a container that may contains files and/or other directories.
Files on the other hand holds information (and cannot contain other files or directories) .

If directoryC is contained in directoryP, then directoryC is a child of directoryP and directoryP is said to be the parent to directoryC.

A directory may have many children, but can only have one parent.

Absolute versus Relative Path¶

The root of the file system is the directory “/” - There is only one root of a directory hierarchy.
A path to a new location (from your current location) can be specified as an absolute path from the root (this will work no matter where we are in the file system):

Navigating relative paths¶

. is the short-form for the current directory
.. signifies the parent directory (akin to pressing Cmnd +↑ on a Mac)
For example, to navigate (i.e. change directories) to the parent directory of the current directory, use the command: cd ..
Note that this command is dependant on your current directory (i.e. the folder you are currently in).

pwd/cd
To print your current working directory type pwd/cd (Mac/Windows) then ENTER .

Clicker question: Absolute versus Relative Path Question¶

Consider this directory hierarchy.

The user is currently in the directory level2
The level1 directory contains a file called test.txt.

Which of the following statements are TRUE?

A relative path to change to directory 301 is ..
Absolute path to test.txt is /Users/ivrbik/301/level1/test.txt
Relative path to test.txt is ../test.txt
Relative path to test.txt is different if user was currently in level3 directory.
There is only one root of the directory hierarchy.

Answer on the sli.do poll in the next cell

display(Markdown('#### Sli.do poll'))

IFrame('https://app.sli.do/event/0oandysw/embed/polls/b97a5644-1486-46be-b5fc-2b65e882baab',
       width=800, height=600)

System Message: WARNING/2 (/home/runner/work/data530_course/data530_course/class/Lecture2.ipynb, line 1)

Non-consecutive header level increase; 0 to 4

Activity: Let’s try to re-create that directory and file structure¶

You will need the following commands:

cd : change directory
pwd : print working directory
mkdir: make directory
ls: list directory contents
touch: create an empty file with a specific name
code/pico/nano/vim: edit the file in a text editor

display(Markdown('#### Supplementary video on absolute vs. relative paths'))

IFrame('https://www.youtube-nocookie.com/embed/sqX6hu7oEew',width="100%",height="500px")

System Message: WARNING/2 (/home/runner/work/data530_course/data530_course/class/Lecture2.ipynb, line 1)

Non-consecutive header level increase; 0 to 4

Break (10 mins)¶

Return at 10:35

The Command Line - Part 2¶

Command Arguments - Mac/Linux/Windows¶

On Mac/Linux arguments are separated by spaces and begin with -

An explanation of arguments can be found by using man then the command name.

Example: man cp (to quit press q)

Commonly used Terminal commands¶

Keep in mind, that in GitBash, the commands should work the same as macOS and Ubuntu

Useful Terminal shortcuts¶

Use the up and down arrows to re-run commands
Ctrl+A to go to the beginning of the line
Ctrl+E to go to the end of the line
To cancel a command, press Ctrl +C .
- The command is canceled, and the command prompt returns. However, any actions performed before the cancel are not undone.

Escape Symbol¶

An escape symbol is used when a command requires input that contains a character with a special meaning. The escape symbol indicates this character is data not part of the command. Windows the caret (^) indicates that whatever character that follows is data rather than part of the command.

Example: cp test.txt a^&b.txt
On macOS/Ubuntu, use the backslash (\).

This is especially common when dealing with spaces in a file name. The other way to handle file names with spaces is to enclose them in double quotes:

cp test.txt "c:\program files\file spaces.txt"

Environment Variable¶

Environment variables allow for customization and control of the command and system environment.

Current variables are seen using the set or env command.

Important variables:

$PATH list of directories where commands/applications will be found
$HOME user home directory

To print out the environment variables, use the echo command:

echo $PATH

Finding Text in Files¶

The grep command allows for searching for text in files that match a pattern (macOS/Ubuntu and find on Windows).

grep stands for “global regular expression print”
Search is case-sensitive (use -i for case-insensitive) and can contain regular expressions.
grep -i will be case-insensitive

Example: the following command searches for ‘er’ in any file that ends in .txt

grep er *.txt

Conclusion¶

The command line is the text interface to the computer that accepts commands that the computer will execute including running programs, manipulating files, and running scripts.

The command line allows for automation and more control than may be available in the user interface. It may also be the only way to interact with the machine if connecting via SSH.

The command environment allows for redirecting the standard input and output using input/output redirection and pipes.

You need to PRACTICE PRACTICE PRACTICE working in the Terminal: to get yourself used to it, I suggest not using the Finder/Explorer on your computer and use the Terminal for everything.
- open will open a file in the default application

Starting and Submitting a Lab on GitHub Classroom and Canvas¶

We didn’t get to this in class so here is a video about the whole process:

IFrame('https://www.youtube-nocookie.com/embed/OrfFTdkghZ4',width="100%",height="500px")

And here it is in text:

Step 1: Accept assignment
- Click link to accept assignment

Step 2: Clone the repo on your local computer
- git clone <https://...>

Step 3: Navigate to that directory

Step 4: Launch a jupyter lab server

Step 5: Make changes and edit .ipynb file

Step 6: Making changes to the README.md
- code README.md and then make changes

Step 7: Commit your changes to the repository
- git add -A or git add .
- git commit -m "This is commit message of some sample code"

Step 8: Pushing the changes above
- git push

Step 9: Copy repo link and paste into Canvas. Submit. After the deadline, we will begin marking submitted assignments.

Summary¶

Briefly explain components of the MDS Install Stack
Define command line and list some of its uses
Explain the purpose of an operating system
Know how to open the command line window on Mac OS and Windows
Enter commands in the Terminal and stop them
Define: file system, folder, file
Explain the difference between an absolute and relative path
Use command line shortcuts to save time
Be able to cancel a command
Explain the reason for an escape symbol
Define and explain the purpose of environment variables. Be able to use grep to search text files.
Describe the full process of submitting a lab on Canvas and GitHub

Code required for Jupyter RISE¶

RISE Template¶

from traitlets.config.manager import BaseJSONConfigManager
from pathlib import Path
path = Path.home() / ".jupyter" / "nbconfig"
cm = BaseJSONConfigManager(config_dir=str(path))
tmp = cm.update(
        "rise",
        {
            "theme": "sky", # https://revealjs.com/themes/
            "transition": "fade",
            "start_slideshow_at": "selected",
            "autolaunch": False,
            "width": "100%",
            "height": "100%",
            "header": "",
            "footer":"",
            "scroll": True,
            "enable_chalkboard": True,
            "slideNumber": True,
            "center": False,
            "controlsLayout": "edges",
            "slideNumber": True,
            "hash": True,
        }
    )

DATA 530

Lecture 2 - Command line and Stack¶

Announcements!¶

Lecture Outline¶

1. Overview of the Platforms used in this program¶

The Command Line - Part 1¶

What is the Command Line?¶

Why learn command line?¶

Why use command line?¶

Command Line on Windows¶

Windows Terminal¶

Mac OS Command Line¶

Entering Commands¶

File System¶

File System¶

Absolute versus Relative Path¶

Navigating relative paths¶

Clicker question: Absolute versus Relative Path Question¶

Activity: Let’s try to re-create that directory and file structure¶

Break (10 mins)¶

The Command Line - Part 2¶

Command Arguments - Mac/Linux/Windows¶

Commonly used Terminal commands¶

Useful Terminal shortcuts¶

Escape Symbol¶

Environment Variable¶

Finding Text in Files¶

Conclusion¶

Starting and Submitting a Lab on GitHub Classroom and Canvas¶

Summary¶

Code required for Jupyter RISE¶

RISE Template¶