========= Linux Lab ========= **While you will be able to do subsequent labs on your Virtual Machine, you must complete this lab in the Computer Science Instructional Lab (CSIL).** Objectives ========== #. Become familiar with the Linux environment #. Learn basic terminal commands and how to work with a text editor #. Learn to run a Python program from the command-line #. Learn the basics of Git and chisubmit #. Learn about file permissions #. Learn about redirection and pipes #. Learn about remote access tools If you can't work through the entire lab, we recommend you work through the last three sections (file permissions, redirection and pipes, and remote access tools) at a later time. We have also marked a few sections as optional, which you are welcome to skip if you are pressed for time. That said, you should ultimately work your way through the entire lab, as it will help you become more comfortable with the Linux environment that you will be using throughout the quarter. Linux ===== Linux is an operating system much like OS X or Windows. It has windows, programs, web browsers, and so on. Files are stored in directories that, in turn, are stored in other directories. Although you can access Linux's features using your mouse, as you perform more and more complex tasks, you will find that using the mouse is ineffective. Linux allows us to interact with the computer entirely through text using a program called the terminal. (Macs provide a similar terminal application, and there are ways to use text-based commands on Windows too. But, Linux provides the lowest barrier to entry.) In this lab you will learn how to use the terminal to perform some basic operations in Linux. You will need these skills for the rest of the course. We show many examples of sample output below. The output you see when you run the commands may vary a bit. For example, most of you are not named "Gustav Martin Larsson". Terminal/Shell ============== On your personal computer, you probably navigate your hard drive by double clicking on icons. While convenient for simple tasks, this approach is limited. For example, imagine that you want to delete all of the music files over 5 MB that you haven't listened to in over a year. This task is very hard to do with the standard double-click interface but is relatively simple using the terminal. On the CSIL machine, click the Application button (at the top left) and type "terminal" in the input box. Click the "terminal" icon to open the terminal window. Alternatively, you can use the keyboard shortcut ``Ctrl-Alt-T`` to open a terminal window. A terminal window will open and you will see text of the form:: username@computer:~$ where ``username`` has been replaced by your CNetID and ``computer`` is the name of the machine you happen to be using. This string is called the prompt. When you start typing, the characters you type will appear to the right of the ``$``. In all subsequent examples in this lab and in the course going forward, we will use ``$`` the prompt rather than the full text ``username@computer:~$`` The program that runs within a terminal window and processes the commands the you type is called a *shell*. We use ``bash``, which is the default shell on most Linux distributions, but there are other popular shells, such as ``ksh``, ``tcsh``, etc. The procedure for completing this lab is as follows. For each section, read through the explanatory text and the examples. Then, try these ideas by doing the exercises listed at the bottom of the section. Navigating the File System -------------------------- Files in Linux are stored in directories/folders, just like in OS X/Windows. Directories can hold files or other subdirectories and there are special directories for your personal files, your Desktop, etc.: +------------------+------------------+-------------------+----------------------------------------+ | Name | Linux | Mac | Windows | +==================+==================+===================+========================================+ | Root directory | / | / | C:\\ | +------------------+------------------+-------------------+----------------------------------------+ | Home directory | /home/username | /Users/username | C:\\Documents and Settings\\username | +------------------+------------------+-------------------+----------------------------------------+ .. image:: img/filesystem.username.svg :align: center :width: 650 :height: 250 All of the computers in the lab are connected through a network file system. Effectively there is one very large shared hard drive. Your files are available from any lab computer and all of our directories live in the same space. The figure above illustrates how Linux organizes the file system. Your own computer might have a slightly different organization (e.g., you might replace ``/`` with ``C:``), but the idea is the same. For the above and from this point forward, consider that the text "username" is replaced with your own actual username, which is just your CNetID. Show Files ---------- The terminal will start in your home directory, ``/home/username/``, which is a special directory assigned to your user account. Any computer that you use in CSIL will automatically connect to your home directory and all files that you created or changed in previous sessions in CSIL will be available to you. Two very useful commands are ``pwd`` and ``ls``: +---------+--------------------------------------------------------------+ | ``pwd`` | Prints your current working directory - tells you where you | | | are in your directory tree. | +---------+--------------------------------------------------------------+ | ``ls`` | Lists all of the files in the current directory. | +---------+--------------------------------------------------------------+ The following is an example using these two commands in a terminal window:: username@computer:~$ pwd /home/username/ username@computer:~$ ls Desktop Documents Downloads Music Pictures Public Templates Videos username@computer:~$ Try these commands yourself to verify that everything looks similar. Notice that the directory path and list of files that you see if you open your home folder graphically are identical to those provided by ``pwd`` and ``ls``, respectively. The only difference is how you get the information and how the information is displayed. Change Directory ---------------- +-------------------+--------------------------------------------------------------+ |``cd `` | change to the directory path-name | +-------------------+--------------------------------------------------------------+ | ``cd ..`` | move up/back one directory | +-------------------+--------------------------------------------------------------+ | ``cd`` | move to your home directory | +-------------------+--------------------------------------------------------------+ How can we move around in the file system? If we were using a graphical system, we would double click on folders and occasionally click the "back" arrow. In order to change directories in the terminal, we use ``cd`` (change directory) followed by the name of the destination directory. (A note about notation: we will use text inside angle brackets, such as ```` as a place holder. The text informally describes the type of value that should be supplied. In the case of ````, the desired value is the path-name for a file. More about path-names later.) For example if we want to change to the ``Desktop`` directory, we type the following in the terminal:: cd Desktop Here is an example of changing to the desktop directory in the terminal. We use ``pwd`` and ``ls`` to verify where we are and where we can go:: username@computer:~$ pwd /home/username/ username@computer:~$ ls Desktop Documents Downloads Music Pictures Public Templates Videos username@computer:~$ cd Desktop username@computer:~/Desktop$ pwd /home/username/Desktop/ username@computer:~/Desktop$ ls username@computer:~/Desktop$ Notice that after we ``cd`` into the ``Desktop`` the command ``pwd`` now prints out:: /home/username/Desktop/ rather than:: /home/username/ In the beginning, there are no files in the Desktop directory, which is why the output of ``ls`` in this directory is empty. We can move up one step in the directory tree (e.g., from ``/home/username/Desktop`` to ``/home/username`` or from ``/home/username`` to ``/home``) by typing ``cd ..`` Here "up" is represented by "``..``" In this context, this command will move us up one level back to our home directory:: username@computer:~/Desktop$ pwd /home/username/Desktop/ username@computer:~/Desktop$ cd .. username@computer:~$ pwd /home/username/ Notice that the current working directory is also shown in the prompt string. +-------------------+--------------------------------------------------------------+ | ``~`` | shortcut for your home directory | +-------------------+--------------------------------------------------------------+ | ``.`` | shortcut for the current working directory | +-------------------+--------------------------------------------------------------+ | ``..`` |shortcut for one level up from your current working directory | +-------------------+--------------------------------------------------------------+ The tilde (~) directory is the same as your home directory: that is, ``~`` is shorthand for ``/home/username``. Here's another useful shorthand: a single dot (``.``) refers to the current directory. Usually when you use ``cd``, you will specify what is called a *relative* path, that is, you are telling the computer to take you to a directory where the location of the directory is described relative to the current directory. The only reason that the computer knows that we can ``cd`` to ``Desktop`` is because ``Desktop`` is a folder within the ``/home/username`` directory. But, if we use a ``/`` at the *beginning* of our path, we are specifying the path relative to the the "root" or top of the file system. For example:: username@computer:~$ pwd /home/username/ username@computer:~$ cd /home/username/Desktop username@computer:~/Desktop$ pwd /home/username/Desktop username@computer:~/Desktop$ cd /home/username username@computer:~$ pwd /home/username These commands achieve the same thing as the ones above: we ``cd`` into ``Desktop``, a folder within our home directory, and then back to our home directory. Paths that start with a ``/`` are known as *absolute paths* because they always lead to the same place, regardless of your current working directory. Running ``cd`` without an argument will take you back to your home directory without regard to your current location in the file system. For example:: username@computer:~/Desktop$ cd username@computer:~$ pwd /home/username To improve the readabilty of our examples, we will use ``$`` as the prompt rather than the full text ``username@computer:~$`` in the rest of this lab and, more generally, in the course going forward. Keep in mind, though, that the prompt shows your current working directory. Setting Up Your CMSC 12100 Directory ------------------------------------ We need a set of files to practice our commands on. Unfortunately, your home directories are mostly empty. In this section, we're going to download a set of files for you to work with. We will do this using Git, a version control system and code-sharing tool. Git will be described in more depth below. For now, please execute the following: - Make sure that your departmental Git account is correctly set up. Using a browser, go to https://mit.cs.uchicago.edu/ and try to log in with your CNetID and password. When you type in your username, use only your CNetID username (without "@uchicago.edu"). If you are unable to log in, please tell a TA. - Back in a terminal window, make sure that you are in your home directory ``/home/username`` using the ``pwd`` command. If you are not in that directory then use ``cd`` to navigate to it. - Run the following command in the terminal if you are on a CSIL computer:: cs-setup-script cmsc12100-aut-18 .. note:: Copy-Paste: In Windows (Mac) you usually copy-paste with ``Ctrl-C`` (``Command-C``) and ``Ctrl-V`` (``Command-V``). These short-cuts are available in graphical programs in Linux but not in the Terminal. Instead you can copy text just by selecting it with your mouse. Select the line that starts with ``cs-setup-script...`` above to copy it. You can paste by clicking the middle mouse button where you would the copied text to go. Middle click in the terminal. You should also be able to use ``Ctrl-Shift-C`` and ``Ctrl-Shift-V``, but you may find that this method does not reliably work. - The setup script will ask you first to enter your CNetID:: Enter your CNetID [username]: Where your CNetID will appear in place of ``username``. You can either type in your CNetID or hit enter to accept the username in brackets. Next, you will be asked for your CNetID password:: Enter your CNetID password: The script needs your password to access your information on the CS department's Git server and will handle it securely. .. note:: You may be accustomed to seeing an asterisk character appear for each password character that you type on a web browser. This will not happen when you type passwords into the terminal. The password is not "echoed" back in any way (not even with asterisks), so don't be alarmed if it looks as if you're password isn't being typed in. - Next, the script will print this:: You are a member of the following repositories. Please select the one you want to use: [1] username [X] Exit Choose one: Choose ``1``. Later in the quarter, you will become a member of other repositories as you work in groups with other students. - If successful, the script will print out the following:: Setting up your Git repository... Your git repository has been created in /home/username/cmsc12100-aut-18-username Setting up chisubmit... chisubmit has been set up. You can use chisubmit commands inside /home/username/cmsc12100-aut-18-username Where, once again, your CNetID will appear in place of ``username``. This output indicates that your Git repository has been correctly set up. The script also configured your ``cmsc12100`` directory for ``chisubmit``, a tool you will use to submit your programming assignments. We'll discuss ``chisubmit`` in Monday's lab. - After running the setup script, list the files in your home directory. You should see a new directory ``cmsc12100-aut-18-username``. This directory will contain all of your work for this class. It contains a subdirectory, ``lab1``, that has some files for us to play with. You will learn how to manipulate these files in the next section. Note that you will also see subdirectories named ``pa0`` and ``pa1``. You can ignore these for now. Use ``pwd``, ``ls``, and ``cd`` to navigate to the ``lab1`` subdirectory. Using an Editor --------------- .. figure:: img/ubuntu-sublime-1.png :align: center :width: 780px :alt: List the files in the ``lab1`` directory. You should see the following:: hello_world.py my_echo.py my-input.txt test.txt How do we view and edit the contents of these files? There are many high-quality text editors for Linux. We will use `Sublime Text `_, which is good for writing code. You can open a specific file, say ``test.txt``, using the ``subl`` command from the Linux command-line by typing:: subl test.txt When you run this command, you will get a new window that displays the following text:: Lab 1 Test file =============== Author: Firstname Lastname If the file is blank, quit ``subl`` and ensure that the file ``test.txt`` exists in your local directory (use ``ls`` to list the files in your local directory). If it does not, use ``cd`` to navigate to the ``lab1`` subdirectory inside the ``cmsc12100-aut-18-username`` directory. For now, we will use sublime (``subl``) in a very basic way. You can navigate to a particular place in a file using the arrow keys and then type typical characters and delete them as you would in a regular text editor. You can save your changes using the ``save`` option in the file menu or use the keyboard shortcut ``Crtl-s``. To quit, you can use the file menu ``quit`` option or the keyboard shortcut ``Ctrl-q``. As an aside, you can also launch Sublime Text from the application launcher: simply click the Application button (at the top left of your screen), type "sublime" in the input box, and a "Sublime Text" icon should appear. You can then use the ``File`` menu to navigate the correct file. Later on you may find it useful to have access to a graphical tree view in your editor. You can access this by launching folder view of sublime rather than just a single file via ``subl .`` Exercises ~~~~~~~~~ Make sure that you are comfortable with this level of usage: #. Add your name after ``Author:`` in this file #. Save the file #. Close and reopen the file in sublime and ensuring that your name is still there #. Finally, close sublime Copy (``cp``), Move (``mv``), Remove (``rm``), and Make Directory (``mkdir``) ----------------------------------------------------------------------------- +---------------------------------+----------------------------------------------+ | ``cp`` | copy the source file to the new destination | +---------------------------------+----------------------------------------------+ | ``mv`` | move the source file to the new destination | +---------------------------------+----------------------------------------------+ | ``rm`` | remove or delete a file | +---------------------------------+----------------------------------------------+ | ``mkdir`` | make a new empty directory | +---------------------------------+----------------------------------------------+ Sometimes it is useful to copy a file. To copy a file, use the command:: cp where ```` is replaced by the name of the file you want to copy and ```` is replaced by the desired name for the copy. An example of copying the file ``test.txt`` to ``copy.txt`` is below:: $ cp test.txt copy.txt ```` can also be replaced with a path to a directory. In this case, the copy will be stored in the specified directory and will have the same name as the source. Move (``mv``) has exactly the same syntax, but does not keep the original file. Remove (``rm``) will delete the file from your directory. If you want to copy or remove an entire directory along with its the files, the normal ``cp`` and ``rm`` commands will not work. Use ``cp -r`` instead of ``cp`` or ``rm -r`` (the ``r`` stands for "recursive") instead of ``rm`` to copy or remove directories: Make sure you want to remove *everything* in the named directory, including subdirectories, *before* you use ``rm -r``. You can make a new directory with ``mkdir directoryname``, where ``directoryname`` is the desired name for the new directory. Exercises ~~~~~~~~~ Try the following tasks to practice and check your understanding of these terminal commands. 1. Execute the above copy command and use ``ls`` to ensure that both files exist. 2. Move the file ``copy.txt`` to the name ``copy2.txt``. Use ``ls`` to verify that this command worked. 3. Make a new directory named ``backups`` using the ``mkdir`` command. 4. Copy the file ``copy2.txt`` to the ``backups`` directory. 5. Verify that step (4) was successful by listing the files in the ``backups`` directory. 6. Now that we have a copy of ``test.txt`` in the backups directory we no longer need ``copy2.txt``. Remove the file ``copy2.txt`` in this directory. It can be hard to spell directory or file names exactly, so the terminal provides an autocomplete mechanism to guide you through your folder explorations. To access this functionality simply start typing whatever name you are interested in the context of a command and then hit tab. If there is only one way to finish that term hitting tab will fill in the rest of the term, for instance, if we typed ``ls b`` and then hit tab it would automatically finish the word ``ls backups`` and then await our hitting enter. If there is MORE than one way to finish a term, like if we had another folder called ``backups-old``, then hitting tab twice with cause the terminal to display all of the options available. Run a Python Program ==================== +----------------------+------------------------------------+ | ``python3 file.py`` | runs the python program file.py | +----------------------+------------------------------------+ In this class, you will learn Python. To run a Python program, use the command ``python3`` and the name of the file that contains your program. Use ``ls`` to verify that there there is a file named ``hello_world.py`` in your ``lab1`` directory. Now, run the program in ``hello_world.py`` by typing (don't forget about autocomplete!):: python3 hello_world.py This program is a very simple. It just prints "Hello, World!" to the screen. .. note:: There are several variants of Python, including Python 2.7 and Python 3. We will be using Python 3 and the corresponding ``python3`` interpreter. The CSIL machines have Python 2.7 installed as the default Python. As a result, the command ``python`` runs a version of Python 2.7. There are some differences between the two languages and Python 3 programs may not run properly using a Python 2.7 interpreter. Edit and Run a Python Program ----------------------------- In this section you will modify and rerun the program in ``hello_world.py``. This change is very simple but goes through all the mechanical steps needed to progam. Open the file ``hello_world.py`` with the command:: subl hello_world.py The file contains a single line of code:: print("Hello, World!") Change this line so that it instead says "Hello " and then your name. For example if your name were Gustav Larsson, the line would read:: print("Hello, Gustav!") Do the following steps: #. Save the file ``hello_world.py`` in sublime (forgetting to save is a surprisingly common error) #. Rerun the program using ``python3`` Let's reinforce the steps to programming in Python with the terminal: #. Change your ``.py`` file with an editor #. Save the file #. Run the file with ``python3`` Forgetting to save the file (step 2) is a very common mistake! Git === Git is a system used for developing software in a group. This system maintains files and all changes that are applied to them. You will each have a personal Git *repository* that is hosted on a central server. The server stores the project files and stores all changes to those files *that have been uploaded to the repository*. We have created accounts and repositories for each of you on a CS department Git server. Also, we will be able to see any changes you upload to your repository, which allows us to provide help remotely, grade your programming assignments, and provide feedback. Git tracks every version of a file or directory using *commits*. When you have made changes to one or more files, you can logically group those changes into a "commit" that gets added to your repository. You can think of commits as "checkpoints" in your work, representing the work you've done since the previous checkpoint. This mechanism makes it possible to look at and even revert to older versions of a file by going back to your code as it was when you "checkpointed" it with a commit. When using Git, your basic working cycle will be: - Log into a CS machine (or your VM) - Change to your ``cmsc12100-aut-18-username`` directory - Download updates from the Git server (we will add files to your repository throughout the quarter). In Git, this operation is called *pulling* from the server. - Work on your files - Create a commit with any changes you have made - Upload the commit to the Git server. In Git, this operation is called *pushing* to the server. The course staff does not have access to any files stored in your home directory or files that you have not pushed to the Git server. All we can access are files that have been pushed to the Git server, so remember to always push your latest commits when you're done or when you ask a question on Piazza that will require us to look at your code. Please navigate to your ``cmsc12100-aut-18-username/lab1`` directory using ``cd``. ``username`` should always be substituted by your CNetID. Creating a commit ----------------- Creating a commit is a two-step process. First, you have to indicate what files you want to include in your commit. Let's say we want to create a commit that only includes the ``hello_world.py`` file that you modified earlier. We can specify this operation explicitly using the ``git add`` command from the Linux command-line:: git add hello_world.py There are various shortcuts that will allow you to add all of the files in a directory, such as ``git add .`` or ``git add --all``. Using these commands is poor practice, because you can easily end up adding files that you did not intend. Instead, it is better to add files explicitly when you create them and then use the command from the Linux command-line: :: git add -u when you want to add any file that has changed since your last commit. To create the commit, use the ``git commit`` command. This command will take all the files you added with ``git add`` and will bundle them into a commit:: git commit -m"Made some changes to hello_world.py" The text after the ``-m`` is a short message that describes the changes you have made since your last commit. Common examples of commit messages might be "Finished part 1 of the homework" or "Finished lab 1". .. note:: If you forget the ``-m"Comment"`` at the end then Git will think that you forgot to specify a commit message. It will graciously open up a default editor so that you can enter such a message. On the CS machines this editor is ``vim``. To escape the vim view, press ``ZZ`` (shift-z twice). Now try ``git commit`` again and don't forget the ``-m"Comment"``. Once you run the above command, you will see something like the following output:: [master 99232df] Made some changes to hello_world.py 1 file changed, 1 insertion(+), 1 deletion(-) You've created a commit, but you're not done yet: you haven't uploaded it to the server yet. Forgetting this step is actually a very common pitfall, so don't forget to upload your changes. You must use the ``git push`` command for your changes to actually be uploaded to the Git server. *If you don't, the graders will not be able to see your code*. Simply run the following command from the Linux command-line:: git push You should see something like this output:: Counting objects: 7, done. Delta compression using up to 16 threads. Compressing objects: 100% (4/4), done. Writing objects: 100% (4/4), 452 bytes, done. Total 4 (delta 1), reused 0 (delta 0) To git@git-dev.cs.uchicago.edu:cmsc12100-aut-18/username.git c8432e4..99232df master -> master You can ignore most of those messages. The important thing is to not see any warnings or error messages. You can verify that our Git server correctly received your commit by visiting the following page:: https://mit.cs.uchicago.edu/cmsc12100-aut-18/username Where ``username`` should be substituted by your CNetID. This URL takes you to the web frontend of our Git server (please note that you will have to log in using your CNetID and password). More specifically, the above URL will show you the contents of your repository, exactly as it appears on the Git server. You can click on "Files" to see your repository's files, and on "Commits" to see the latest commits uploaded to the server. If you see a commit titled "Made some changes to Hello World", then your commit was successfully uploaded. In general, if you're concerned about whether the graders are seeing the right version of your code, you can just go to the above URL. Whatever is shown on that page is what the graders will see. If you wrote some code, and it doesn't show up in the above URL, make sure you didn't forget to add your files, create a commit, and push the most recent commit to the server. Pulling changes from "upstream" ------------------------------- When we distribute new homework assignments or lab materials, we will do so through Git. These files are located in a separate repository on our Git server, which we call the "upstream" repository. The setup script you ran earlier already configured your Git repository so you can easily download any new files we upload to the upstream repository). To download these changes, run this command from inside the ``cmsc12100-aut-18-username`` directory:: git pull upstream master If you run it now, nothing will actually happen, since we haven't changed anything in "upstream" since the start of this lab. You should see something like this output:: From git-dev.cs.uchicago.edu:cmsc12100-aut-18/cmsc12100-aut-18 * branch master -> FETCH_HEAD Already up-to-date. When you pull from "upstream", Git automatically downloads any new files or changes that have been committed to "upstream" and updates the files in your repository. If you have made local changes to files that have changed upstream, Git will attempt to merge these changes. After you've pulled from upstream, any new files or changes will only be downloaded to your local copy of ``cmsc12100-aut-18-username``. As with any other changes to your code, you need to run ``git push`` to upload them to the Git server (you don't need to do a ``git commit`` to prepare a commit, though; ``git pull`` already takes care of this task). - Every time you work on your code, you should run ``git pull upstream master`` in your ``cmsc12100-aut-18-username`` directory before you do anything else. Sometimes, the instructors notice typos or errors in the code provided for a programming assignment, and they'll commit fixes to upstream. By running ``git pull upstream master``, you can make sure that those fixes propagate to your code too. Pulling your changes from the server ------------------------------------ If you have done work and committed it to the server from a lab computer and now wish to work on your VM (or vice versa), you will need to pull these changes from the server to your VM. To download these changes, run this command from inside the ``cmsc12100-aut-18-username`` directory:: git pull It is important that you commit your changes after every session and that you pull from both ``upstream`` and ``cmsc12100-aut-18-username`` before you start to do any work. .. note:: Your output may vary from our sample output slightly. Do not worry about the difference unless you see an error message or a warning message. ``git add`` revisited and ``git status`` ---------------------------------------- So far, we've created a single commit with a single file that we had already supplied in the ``lab1`` directory. If you create new files, Git will not consider them a part of the repository. You need to add them to your repository explicitly. For example, let's create a copy of ``hello_world.py``:: cp hello_world.py hello_universe.py Is ``hello_universe.py`` part of your repository? You can use the following command to ask Git for a summary of the files it is tracking:: git status This command should output something like this:: # On branch master # Changes not staged for commit: # (use "git add ..." to update what will be committed) # (use "git checkout -- ..." to discard changes in working directory) # # modified: test.txt # # Untracked files: # (use "git add ..." to include in what will be committed) # # hello_universe.py no changes added to commit (use "git add" and/or "git commit -a") The exact output may vary depending on how far along you got in previous sections of the lab. However, the important thing is that there are two types of files listed here: - ``Changes not staged for commit``: This is a list of files that Git knows about and have been modified since your last commit, but which have not been added (with ``git add``). - ``Untracked files``: This is a list of files that Git has found in the same directory as your repository, but which Git isn't keeping track of. You may see some automatically generated files in your ``Untracked files`` section. Files that start with a pound sign (#) or end with a tilde should *not* be added to your repository. Files that end with a tilde are backup files created by some editors that are intended to help you restore your files if your computer crashes. In general, files that are automatically generated should not be committed to your repository. Other people should be able to generate their own versions, if necessary. To add a previously untracked file to your repository, you can just use ``git add`` (unlike the previous commands, don't actually run this just yet; you will be doing a similar exercise later on):: git add hello_universe.py If you re-ran ``git status`` you would see something like this:: # On branch master # Changes to be committed: # (use "git reset HEAD ..." to unstage) # # new file: hello_universe.py # # Changes not staged for commit: # (use "git add ..." to update what will be committed) # (use "git checkout -- ..." to discard changes in working directory) # # modified: test.txt Notice how there is now a new category of files: ``Changes to be committed``. Adding ``hello_universe.py`` not only added the file to your repository, it also staged it into the next commit (which, remember, won't happen until you actually run ``git commit``). The ``git status`` command reports the status on the local copy of the full repository. If you wish to look at the status of a smaller part of the repository (the directory you are working in for example), you can add a path name to the status command. For example:: git status . reports the status of the current directory (a single dot is the path used to refer to the current directory). Unstaging, discarding changes, and removing files (optional, skip if short on time) ------------------------------------------------------------------------------------ Take a closer look at the ``git status`` output above. Git is providing you hints in case you want to undo some of your work. For example, you can use ``git reset hello_universe.py`` to unstage the file. Doing so reverses ``git add hello_universe.py`` so you can create a commit only of changes to other files. This is good practice if you think the changes you made to ``hello_universe.py`` don't logically go in the commit you are going to make. Another useful git command is ``git checkout``. This command will undo modifications to files. If you again look at the above ``git status`` output, you will see in the last line that test.txt was modified. To undo any changes to the file, type ``git checkout test.txt``. This command will revert the file content to match the last commit you made in your repository's history Finally, if you would like to remove a file from your directory, using ``git rm test.txt`` combines the result of doing ``rm test.txt`` and ``git add test.txt``. Looking at the commits log (optional, skip if short on time) -------------------------------------------------------------- Once you have made multiple commits, you can see these commits, their dates, commit messages, author, and an SHA-1 hash by typing ``git log``. This command will open a scrollable interface (using the up/down arrow keys) that you can get out of by hitting ``q``. Exercises --------- #. You have already changed the ``test.txt`` file in your directory. Verify this by using the command ``git status``. You should see it under ``Changes not staged for commit``. #. Use ``git add`` and ``git commit`` to create a commit that includes only the ``test.txt`` file. A good commit message would be "Added my name as Author in test.txt". #. Upload your work to the server using ``git push``. #. Verify that this file was sent by again using the command ``git status``. You should see that the file ``test.txt`` is no longer listed. #. If you have not already done so, use ``cp`` to make a copy of ``hello_world.py`` named ``hello_universe.py``. #. If you run ``git status``, ``hello_universe.py`` should show up under ``Untracked files``. Add it to the repository using ``git add``. #. Run ``git status`` again. Is ``hello_universe.py`` in a different category of files now? #. Although we have added this file, we have not yet created a commit. Create a commit and push it to the server. #. Run ``git status`` a final time to verify that ``hello_universe.py`` was committed (if so, you should not see it in any category of files) #. Run ``git push`` to upload your changes to the server. **We strongly recommend you to check in and push changed files as often as possible, especially if you finished some work and are about to log off a computer. This way the latest files are accessible from any other computer where your repository is set up.** chisubmit ========= You will be using a locally-developed system named ``chisubmit`` to submit your programming assignments. The set-up script that you ran earlier set you up to use ``chisubmit`` in addition to initializing your Git repository. All chisubmit commands should be run from within your ``cmsc12100-aut-18-username`` directory. ``chisubmit`` has commands for managing assignments. Here are descriptions and sample runs of some of the more useful commands. You can run these commands as you read through this section. ``chisubmit student assignment list``: lists upcoming programming assignments and their deadlines. :: username@cs-vm:~/cmsc12100-aut-18-larsson$ chisubmit student assignment list pa0 2018-10-07 23:59:59-05:00 Programming Assignment #0 ``chisubmit student assignment show-deadline ``: lists deadline information for the specified programming assignment. :: username@cs-vm:~/cmsc12100-aut-18-larsson$ chisubmit student assignment show-deadline pa0 Programming Assignment #0 Now: 2018-09-28 09:51:41-05:00 Deadline: 2018-10-07 23:59:59-05:00 The deadline has not yet passed You have 9 days, 14 hours, 8 minutes, 18 seconds left ``chisubmit student assignment register ``: registers a student for a specific assignment. You will do this step *once* per assignment. :: username@cs-vm:~/cmsc12100-aut-18-larsson$ chisubmit student assignment register pa0 Your registration for pa0 (Programming Assignment 0) is complete. ``chisubmit student assignment submit pa0``: submits your current commit :: username@cs-vm:~/cmsc12100-aut-18-larsson$ chisubmit student assignment submit pa0 SUBMISSION FOR ASSIGNMENT pa0 (Programming Assignment 0) -------------------------------------------------------- This is an INDIVIDUAL submission for Gustav Martin Larsson The latest commit in your repository is the following: Commit: eeed8efa66a13c0b04c587acdda43fbe75c9b99b Date: 2018-09-28 13:47:16-05:00 Message: Added log for testing purposes Author: Gustav Martin Larsson PLEASE VERIFY THIS IS THE EXACT COMMIT YOU WANT TO SUBMIT You currently have 2 extensions You are going to use 0 extensions on this submission. You will have 2 extensions left after this submission. Are you sure you want to continue? (y/n): y Your submission has been completed. ``chisubmit`` has many other commands, including command for canceling registrations, canceling submissions etc. You can find detailed instructions on these and other commands `here `__. More Git (optional, skip if short on time) ========================================== Pulling on another machine -------------------------- Now let's return to Git. This part requires that you have a laptop with you and that you have gone through the process of setting-up your laptop with the CS121 VM and with your CS121 repository. If you do not have a laptop or do not have one with you, you can just skip this this section. If you look at your files in ``lab1`` on your VM, you'll notice that they are out of date. The changes you made to ``test.txt`` are not visible and ``hello_universe.py``, the new file you added earlier, is missing. Why? Because your local copy on your VM is out of sync with the authoritative version of repository on the server. Run ``git pull`` from the Linux command-line to pick up your changes. Once you run this command the file ``test.txt`` should have changed and there should be a file named ``hello_universe.py``. As long as you committed and pushed all your changes on the CSIL machines to the server, the copies of your repository on the CSIL/CS machines, your VM, and the server will be the same. Merge conflicts --------------- The beauty of Git specifically, and version control in general, is that you can share repositories with other people and you can work on the code separately. Merge conflicts arise when different copies of the repository get changed in incompatible ways. Unfortunately, this complication can arise even if you are the only one working on your repository! You just need to work on your code using multiple machines. Let's work through an example. Let's say I take the following steps using a CSIL machine: 1. Change the line in ``hello_world.py`` from: :: print("Hello, World!") to: :: print("Hello, Chicago!") 2. Add and commit these changes. 3. Forget to do a push. And then switch to my VM and do the following: 1. Run ``git pull`` inside my cs121 directory to pick up the most recent copy from the server. #. Change the line in ``hello_world.py`` from: :: print("Hello, World!") to: :: print("Hello, New York!") (Notice that the change from "World" to "Chicago" did not make it to my VM because I forgot to push!) 2. Add, commit, and push these changes to the server. And then, finally, switch back to my CSIL machine and run ``git pull`` to pick up the most recent version from the server. This command will fail with an error like the following: :: username@cs-vm:~/cmsc12100-aut-18-larsson/lab1$ git pull remote: Counting objects: 4, done. remote: Compressing objects: 100% (3/3), done. remote: Total 4 (delta 1), reused 0 (delta 0) Unpacking objects: 100% (4/4), done. From mit.cs.uchicago.edu:cmsc12100-aut-18/larsson 62c72de..b70ae2a master -> origin/master Auto-merging lab1/hello_world.py CONFLICT (content): Merge conflict in lab1/hello_world.py Automatic merge failed; fix conflicts and then commit the result. Git was not able to reconcile my local changes with the version on the server automatically and so, it updates ``hello_world.py`` to reflect the conflicts. The file ``hello_world.py`` will look like this: .. code:: <<<<<<< HEAD print("Hello, Chicago!") ======= print("Hello, New York!") >>>>>>> b70ae2a739c7775189b284be04ae568568ac3c62 The lines between ``<<<<<<< HEAD`` and ``=======`` contain the code as it exists on the CSIL machine, where as the lines between ``=======`` through ``>>>>>>> b70ae2a739c7775189b284be04ae568568ac3c62`` contain the code from the server. In general, a failed merge can yield a mix of merged and unmerged blocks. I will need to resolve these conflicts by choosing among the offending lines and removing the conflict markers (``<<<<<<< HEAD``, ``=======``, and ``>>>>>>> b70ae2a739c7775189b284be04ae568568ac3c62``). And then I'll need to add, commit, and push the updated files, before I can pull or push my code successfully. .. note:: The sections below expand on useful Linux concepts and skills, but you can work through them at a later time if you want (including on your VM, instead of on a CSIL machine). File Permissions ================ Sometimes we want to restrict who can access certain resources on the file system. Most file systems assign 'File Permissions' (or just permissions) to specific users and groups of users. Unix is no different. File permissions dictate who can read (view), write (create/edit), and execute (run) files on a file system. All directories and files are owned by a user. Each user can be a member of one or more groups. To see your groups, enter the command 'groups' into the command line. File permissions in Unix systems are managed in three distinct scopes. Each scope has a distinct set of permissions. **User** - The owner of a file or directory makes up the *user* scope. **Group** - Each file and directory has a group assigned to it. The members of this group make up the *group* scope. **Others** - Every user who does not fall into the previous two scopes make up the *others* scope. If a user falls into more than one of these scopes, their effective permissions are determined based on the first scope the user falls within in the order of user, group, others. Users that fall into each scope can have three specific permissions. **read** - The read permission allows a user to view a file's contents. When set for a directory, this permission allows a user to view the names of files in the directory, but no further information about the files in the directory. ``r`` is shorthand for read permissions. **write** - The write permission allows a user to modify the contents of a file. When set for a directory, this permission allows a user to create, delete, or rename files. ``w`` is shorthand for write permissions. **execute** - The execute permission allows a user to execute a file (or program) using the operating system. When set for a directory, this permission allows a user to access file contents and other information about files within the directory (given that the file has the proper permissions for the user to access it). The execute permission does not allow the user to list the files inside the directory unless the read permission is also set. ``x`` is shorthand for execute permissions. Each permission has a unique value: read = 4, write = 2, execute = 1. As a result, you can describe the permissions of each scope using the sum of its permissions' values. For example, if a file has read and write permissions for the user scope, its permissions can be described as 6 (4 + 2 = 6). Additionally, you can describe a file's permissions using these values for each scope. For example, 761 describes the permissions for a file with read, write, and execute permissions for the user scope, read and write permissions for the group scope, and only execute permissions for the others scope. To list information about a file, including its permissions, type:: ls -l You'll get output of the form:: 1 owner group For example, if we want information on ``/usr/bin/python3.5``:: $ ls -l /usr/bin/python3.5 -rwxr-xr-x 1 root root 3709944 Oct 14 2015 /usr/bin/python3.5 First thing we can notice is that the owner of the file is a user named ``root``. (FYI, ``root`` is a name for an account that has access to *all* commands and files on a Linux system. Other accounts may also have "root" privileges.) The file's group is also ``root``. The permissions are ``-rwxr-xr-x``. These permissions are listed in user, group, and others order. In this example, the owner, ``root``, can read (``r``), write (``w``), and execute (``x``) the file. Users in the ``root`` group and all other users can read and execute the files. Exercises --------- By default, any files or directories that you create will have your username as both the user and the group. (If you run ``groups``, you'll notice that there is a group with the same name as your username. You are the only member of this group.) On our Linux machines, new files are given, by default, to give read and write permissions to user and group and no permissions to other. New directories will be set to have read, write and execute permissions for user and group. #. Verify this claim by running ``ls -l backups/copy2.txt`` and ``ls -ld backups`` in your ``lab1`` directory. The ``-d`` flag tells ``ls`` to list the directory, instead of its contents. Notice that that the first letter in the permissions string for ``backups`` is a `d`, which tells us that ``backups`` is directory. A regular file would have a ``-`` in that spot. Once you have verified the claim, go ahead and remove the ``backups`` directory using the command: ``rm -r backups``. Changing Permissions, Owner, & Group ------------------------------------ +-----------------------------------------+----------------------------------------------+ | ``chmod`` | set the permissions for a file/directory | +-----------------------------------------+----------------------------------------------+ | ``chmod`` | update the permissions for a file/directory | +-----------------------------------------+----------------------------------------------+ | ``chown`` | change the owner of a file to username | +-----------------------------------------+----------------------------------------------+ | ``chgrp`` | change the group of a file | +-----------------------------------------+----------------------------------------------+ | ``cat`` | print the contents of a file to the terminal | +-----------------------------------------+----------------------------------------------+ To change permissions, we use the ``chmod`` command. This command can be used in two ways. We can set the permissions for a file using a 3 digit number (see above) or by adding to and/or removing permissions from the current settings. We can demonstrate this using the ``cat`` command to print file contents to the terminal (we'll make use of an unfamiliar operator, >, but bare with us):: $ echo "Hello!" > testfile $ ls -l testfile -rw-rw---- 1 username username 7 Aug 23 11:22 testfile $ cat testfile Hello! $ chmod 222 testfile #set only write permissions for all scopes $ ls -l testfile --w--w--w- 1 username username 7 Aug 23 11:22 testfile $ cat testfile cat: testfile: Permission denied $ chmod u+r testfile #give user scope read permissions In this last example, we have added user read permissions to ``testfile``. To change the owner of a file or directory (if you are the owner or root), use the command:: chown To change a file's group (if you are the owner or root), use the command:: chgrp It is unlikely that you will need to use these two commands for this course. Exercises ~~~~~~~~~ #. Run ``echo "Hello!" > testfile`` to construct testfile. Look at the permissions using ``ls -l``. #. Change the permissions on ``testfile`` to allow and read access for others. Run ``ls -l testfile`` to check the new permissions. #. Remove user and group write access from ``testfile``. Check the corrected permissions. #. Remove ``testfile`` using ``rm``. Other Shell Features ==================== Wild Cards (using an asterisk) ------------------------------ Sometimes when we enter a string, we want part of it to be variable, or a wildcard. A common task is to list all files that end with a given extension, such as ``.txt``. The wildcard functionality, through an asterix, allows to simply say:: $ ls *.txt The wildcard can represent a string of any length consisting of any characters - including the empty string. It is important to be **careful** using wildcard, especially for commands like ``rm`` which cannot be undone. A command like:: $ rm * ### DO NOT RUN THIS COMMAND! will delete **all** of the files in your working directory! Exercises --------- #. Navigate to your ``cmsc12100-aut-18-username`` directory. What do you see when you run ``ls pa*``? What about ``ls pa*/*``? #. What do you expect to see when you run the command ``ls ../pa*`` from within your ``cmsc12100-aut-18-username/lab1`` directory? Environment Variables --------------------- +------------------------------------+------------------------------------------------------+ | ``printenv`` | print the current state of the environment variables | +------------------------------------+------------------------------------------------------+ | ``export ="some value"`` | define an environment variable | +------------------------------------+------------------------------------------------------+ Sometimes, when we have text (a path, for example) that we use often, we give it a name by assigning it to a variable for convenience. The command:: $ printenv gives a long list of defined variables. Try it in your terminal to see what happens! When we want to define new variables, we use the ``=`` operator and ``export`` command. As you found from using printenv, is typical to name environment variables in all capital letters, so we could define a new variable as simply:: $ NEWVARIABLE=~/Desktop It is important **not to add spaces**; the commands:: $ NEWVARIABLE =~/Desktop $ NEWVARIABLE= ~/Desktop $ NEWVARIABLE = ~/Desktop would all be misinterpreted by the terminal; spaces within quotes are allowed, if accurate. We add a ``$`` before the name of an environment variable in a command to use its value:: $ NEWVARIABLE=~/Desktop $ cd NEWVARIABLE bash: cd: NEWVARIABLE: No such file or directory $ cd $NEWVARIABLE $ pwd /home/username/Desktop A variable created as above is only available to the current shell. It is a local variable, so future shells (such as those that you create when you open a new terminal window using ``Ctrl-Shift-N``) will not have access to it. In order to save variables, we need to export them using the export command. Variables that are exported are called environment variables, which are generally declared as follows:: $ export NEWVARIABLE=~/Desktop For this environment variable to take effect on all future shells, the ``export`` statement needs to be added to a special file called ``.bashrc`` in your home directory (this file gets run every time you start a new shell) Exercises ~~~~~~~~~ #. Define an environment variable and use ``echo``, which takes a list of values as command-line arguments and echos them to the screen, to see its value. #. Open a new terminal window and use ``echo`` in that window to see whether the variable is still defined. Note that environment variables defined at the command-line are only available until you log out. They will not be available the next time you log in. Man Pages --------- A man page (short for manual page) documents or describes topics applicable to Linux programming. These topics include Linux programs, certain programming functions, standards, and conventions, and abstract concepts. To get the man page for a Linux command, you can type:: man So in order to get the man page for ``ls``, you would type:: man ls This command displays a man page that gives information on the ``ls`` command, including a description, flags, instructions on use, and other information. Each man page has a description. The ``-k`` flag for ``man`` allows you to search these descriptions using a keyword. For example:: man -k printf This searches all the descriptions for the keyword ``printf`` and prints the names of the man pages with matches. Running Commands Sequentially ----------------------------- It is often convenient to chain together commands that you want to run in sequence. For example, recall that to print the working directory and list all of the files and directories contained inside, you would use the following commands:: $ pwd /home/username/ $ ls Desktop Documents Downloads Music Pictures Public Templates Videos You could also run them together, like so:: $ pwd ; ls /home/username/ Desktop Documents Downloads Music Pictures Public Templates Videos First, ``pwd`` is executed and run to completion, and then ``ls`` is executed and run to completion. The two examples above are thus equivalent, but the ability to run multiple commands together is a small convenience that could save you some time if there is a group of commands that you want to execute sequentially. .. note:: The shell doesn't care about white space, so it will run any of the following as well:: $ pwd;ls $ pwd ;ls $ pwd; ls $ pwd ; ls Useful Keyboard Shortcuts ------------------------- Used at the Linux prompt, the keyboard shortcut ``Ctrl-P`` will roll back to the previous command. If you type ``Ctrl-P`` twice, you will roll back by two commands. If you type ``Ctrl-P`` too many times, you can use ``Ctrl-N`` to move forward. Here are few more useful shortcuts: - ``Ctrl-A`` will move you to the beginning of a line. - ``Ctrl-E`` will move you to the end of a line. - ``Ctrl-U`` will erase everything from where you are in a line back to the beginning. - ``Ctrl-K`` will erase everything from where you are to the end of the line. - ``Ctrl-l`` will clear the current terminal (your env variables will still be there, but the text display will be gone) - ``Ctrl-shift-t`` will start a new terminal tab - ``Ctrl-shift-w`` will close the current terminal tab Play around with these commands. Being able to scroll back to, edit, and then rerun previously used commands saves time and typing! Redirection =========== The examples in this section will use commands that we've not yet discussed. Refer to the man pages for information about unfamiliar commands. As we already know, commands like ``pwd``, ``ls``, and ``cat`` will print output to screen by default. Sometimes, however, we may prefer to write the output of these commands to a file. In Linux, we can redirect the output of a program to a file of our choosing. This operation is done with the ``>`` operator. Try the following example and compare your output with ours:: $ cd $ touch test-0.txt $ ls > test-1.txt $ cat test-1.txt Desktop Documents Downloads Music Pictures Public Templates test-0.txt test-1.txt Videos $ echo "Hello World!" > test-2.txt $ cat test-2.txt Hello World! $ cat test-2.txt > test-1.txt; cat test-1.txt Hello World! $ rm test-* Two important things to note: #. If you redirect to a file that does not exist, that file will be created. #. If you redirect to a file that already exists, the contents of that file will be overwritten. You can use the append operator (``>>``) to append the output of command to the end of an existing file rather than overwrite the contents of that file. Not only can we redirect the output of a program to a file, we can also have a program receive its input from a file. This operation is done with the ``<`` operator. For example:: $ python3 my_echo.py < my-input.txt In general, all Linux processes can perform input/output operations through, at least, the keyboard and the screen. More specifically, there are three 'input/output streams': standard input (or ``stdin``), standard output (or ``stdout``), and standard error (or ``stderr``). The code in ``my_echo.py`` simply reads information from ``stdin`` and writes it back out to ``stdout``. The redirection operators change the bindings of these streams from the keyboard and/or screen to files. We'll discuss ``stderr`` later in the term. Piping ------ In addition to the ability to direct output to and receive input from files, Linux provides a very powerful capability called piping. Piping allows one program to receive as input the output of another program, like so:: $ program1 | program2 In this example, the output of program1 is used as the input of program2. Or to put it more technically, the ``stdout`` of ``program1`` is connected to the ``stdin`` of ``program2``. As another more concrete example, consider the ``man`` command with the ``-k`` option that we've previously discussed. Let's assume that you hadn't yet been introduced to the ``mkdir`` command. How would you look for the command to create a directory? First attempts:: $ man -k "create directory" create directory: nothing appropriate $ man -k "directory" (a bunch of mostly irrelevant output) As we can see, neither of these options is particularly helpful. However, with piping, we can combine ``man -k`` with a powerful command line utility called ``grep`` (see man pages) to find what we need:: $ man -k "directory" | grep "create" mkdir (2) - create a directory mkdirat (2) - create a directory mkdtemp (3) - create a unique temporary directory mkfontdir (1) - create an index of X font files in a directory mklost+found (8) - create a lost+found directory on a mounted Linux second extended fil... mktemp (1) - create a temporary file or directory pam_mkhomedir (8) - PAM module to create users home directory update-info-dir (8) - update or create index file from all installed info files in directory vgmknodes (8) - recreate volume group directory and logical volume special files Nice. Exercises ~~~~~~~~~ #. Use piping to chain together the ``printenv`` and ``tail`` commands to display the last 10 lines of output from ``printenv``. #. Replicate the above functionality without using the ``|`` operator. (hint: Use an intermediate file.) Remote Access ============= We'll finish up with a description of some useful commands. If you run out of time, you can skip this part and return to it later. There are two main tools for accessing a remote computer through the command line: one for running commands on the remote computer, and one for file transfer. The first of these commands is much more likely to be useful in this class. SSH --- SSH allows you to open a terminal session on a computer remotely, and is a major motivation for becoming proficient with the terminal. The following command:: $ ssh username@domain begins an SSH session, and allows you to access all of your files and programs on the remote computer (as long as these programs can be executed through the shell). The command for SSHing into CSIL Linux computers is:: $ ssh CNETID@linux.cs.uchicago.edu You should try this now and ask a question if you have trouble, as it is something you may have to do for your CS classes here. To exit an SSH session, simply use the command ``exit``. ``ssh`` is installed by default on Linux and OSX. `PuTTY `__ is a popular SSH client for Windows. SCP --- While SSH allows you to log in to another computer, SCP provides the ability to transfer files between computers. ``scp`` is useful, but do **not** use it to move files in your ``cmsc12100-aut-18`` repository between machines. It is much safer to use Git to manage the files in your repository. In general, SCP is called as:: $ scp user@host1:/path/to/file1 user@host2:path/to/file2 and copies a file from one computer (the first argument) and places it in the second computer (the second argument). If you want to copy a file to your local computer, you can simply specify the second argument as a file path, without the username or domain name; the same principle applies for copying from your local computer. If you want to leave the file named as it was, you don't need to specify the file name in the second argument. An example of using SCP would be:: $ ls Desktop Downloads $ scp username@linux.cs.uchicago.edu:~/cmsc12100/assignment1/Grade.txt . $ ls Desktop Downloads Grade.txt Recall that a single dot (``.``) refers to the current directory. Final Notes =========== Sometimes, a program will run indefinitely or misbehave. When this happens, you can type ``Ctrl-C`` to send an interrupt signal to the running program, which usually causes it to terminate. On occasion, you may need to type ``Ctrl-C`` a few times. Typing ``Ctrl-D`` sends an end of input signal, which tells the program that no more information is coming.