Remote Access Protocols

I'll now take a moment and talk about the various networking protocols that you can use to communicate with a Git remote. There are four available protocols: Local, Git, SSH, and HTTP.

$ git clone /var/git/repos/myproj.git

$ git clone git://repos/myproj.git

$ git daemon --help

SSH

Secure Shell (SSH) is a more commonly used protocol. It is also one that most users and network administrators are familiar with; it is also inexpensive to set up and use.

If you're not familiar with SSH, it operates on the idea of authenticated access using private and public keys. The public key goes on the resource you need access to (the Git remote in this case), and the private key goes into a special ˜/.ssh subdirectory on your local machine. As long as the private key on the user's system corresponds to a public key on the desired resource, the user can connect and transfer information without having to log in each time. Authentication is handled using the keys.

There are two forms of paths that you can use with SSH:

$ git clone ssh://<username>@mygitserver.mycompany.com/myproject.git

or

$ git clone <username>@mygitserver.mycompany.com:myproject.git

Figure 12.2 illustrates the difference between the traditional login methods of access (top) and SSH access with keys (bottom).

$ git clone https://mycompany.com/repos/myproject.git

The Remote Command

The git remote command allows you to manage your connections and interactions with remote repositories. The syntax is as follows:

git remote [-v | --verbose]
git remote add [-t <branch>] [-m <master>] [-f] [--[no-]tags] [--mirror=<fetch|push>] <name> <url>
git remote rename <old> <new>
git remote remove <name>
git remote set-head <name> (-a | --auto | -d | --delete | <branch>)
git remote set-branches [--add] <name> <branch>…
git remote get-url [--push] [--all] <name>
git remote set-url [--push] <name> <newurl> [<oldurl>]
git remote set-url --add [--push] <name> <newurl>
git remote set-url --delete [--push] <name> <url>
git remote [-v | --verbose] show [-n] <name>…
git remote prune [-n | --dry-run] <name>…
git remote [-v | --verbose] update [-p | --prune] [(<group> | <remote>)…]

As you can see, there are a lot of options here, but you will probably only use a few of them. Many of these options relate to the idea of a remote reference that I talked about earlier in this chapter; this is a short name that refers to the longer protocol string (https://…, git://…, ssh://…) that is the actual location of the remote repository.

Let's look at a couple of the common use cases for the git remote command. As I mentioned, when you clone down a remote repository, Git automatically sets up a remote (reference) named origin that maps to whatever location you cloned from. You can see what a remote maps to at any point by using the -v or --verbose option of the command. Here is an example:

$ git clone https://github.com/brentlaster/gradle-greetings

$ cd gradle-greetings

$ git remote -v
origin  https://github.com/brentlaster/gradle-greetings (fetch)
origin  https://github.com/brentlaster/gradle-greetings (push)

Now, for any of the commands that I need to use to interact with the remote repository at http://github.com/brentlaster/gradle-greetings, I can just use origin instead of typing out that longer path.

I can also add, remove, and rename remote references. Again, these are just references or aliases. Modifying these references does not affect the remote repository.

git remote add version2 https://github.com/brentlaster/gradle-greetings-sets

$ git remote -v
origin  https://github.com/brentlaster/gradle-greetings (fetch)
origin  https://github.com/brentlaster/gradle-greetings (push)
version2        https://github.com/brentlaster/gradle-greetings-sets (fetch)
version2        https://github.com/brentlaster/gradle-greetings-sets (push)

$ git remote rename version2 origin2

$ git remote -v
origin  https://github.com/brentlaster/gradle-greetings (fetch)
origin  https://github.com/brentlaster/gradle-greetings (push)
origin2 https://github.com/brentlaster/gradle-greetings-sets (fetch)
origin2 https://github.com/brentlaster/gradle-greetings-sets (push)

$ git remote rm origin2

$ git remote -v
origin  https://github.com/brentlaster/gradle-greetings (fetch)
origin  https://github.com/brentlaster/gradle-greetings (push)

Notice that there are also options to get and set the URL associated with the remote. This can be useful if you cloned with one protocol but want to use a different one now.

Now that you understand the basic idea of remotes, let's look at how Git interacts with them and how you keep track locally of where things are on the remote side.

How Git Interacts with the Remote Environment

Remote repositories in Git are dumb. This doesn't mean they're a bad idea, but rather that they don't have to do a lot of processing. So, you could say that the model around them is smart.

Git does not maintain a constant connection from the local repository to the remote repository. (If it did and it required this, you wouldn't be able to do disconnected development.) Rather, Git checks in with the remote repository to get updated status information and contents. This checking in occurs whenever a Git command that requires interaction with the remote repository is initiated in the local environment—essentially, when doing a fetch, pull, or push operation.

For the necessary duration of those operations, Git in the local environment establishes the connection to the remote repository, gathers information about where branches of interest are in the remote repository, and updates or downloads content as appropriate.

So, Git can get by with just temporary connections to the remote environment. This is similar to how most traditional source management systems work. A connection to the server is only established and used when updating content between the server and the local working area.

Remote Tracking Branches

Having the connection between the local environment and the remote environment as a temporary one, only activated when an operation demands, is a useful construct for keeping things simple. It also enables you to limit overhead and bandwidth, and support paradigms like disconnected development.

However, it is not as useful if you are working in your local environment and you want to know how your local changes compare to the versions of changes in the remote repository (status) or you want to pull content from the remote repository but not merge it in yet. These kinds of operations require some persisted knowledge of the state of the remote repository (between operations that establish physical connections to the remote environment).

Git persists this information about the state of the remote repository by setting up remote tracking branches in the local repository. Essentially, the local repository contains state information about the remote repository based on the last time you talked with it (connected to the remote).

These copies of the remote branches exist alongside the local branches in the local repository. As such, there has to be a way to distinguish the two branch types because you will usually have local branches with the same name as remote branches. For example, I will have a master branch on the remote and a master branch in the local repository.

To help distinguish the remote tracking branches from the local branches in the local repository, remote tracking branches have a namespace associated with them—the remote reference name, such as origin, origin2, and so on. For example, if your remote reference to the area you cloned from is called origin, then the remote tracking branch for master in the local repository will be named origin/master (or more accurately, remotes/origin/master). The local master branch will just be master. Table 12.1 summarizes the characteristics of the various branches.

Let's see how this all fits in with the Git model by looking at the clone command.

Git Clone

As I discuss in the early chapters, cloning is how you start working with an existing Git repository. The idea is that you get a copy of the repository from the server (remote side) down to your local environment (a local directory).

You initiate this copy with the git clone command. The syntax is as follows:

git clone [--template=<template_directory>]
          [-l] [-s] [--no-hardlinks] [-q] [-n] [--bare] [--mirror]
          [-o <name>] [-b <name>] [-u <upload-pack>] [--reference <repository>]
          [--dissociate] [--separate-git-dir <git dir>]
          [--depth <depth>] [--[no-]single-branch]
          [--recursive | --recurse-submodules] [--] <repository>
          [<directory>]

An example clone command is

$ git clone ssh://git@gitserver:repository

or

$ git clone http://gitserver/repository

When you do a clone, several things take place, including the following:

• The .git repository is copied from the server (remote side) to the directory locally. This instantiates a new local environment.
• Git creates remote tracking branches in the clone that correspond to all of the branches in the remote repository.
• For the currently active HEAD in the remote, Git creates a corresponding local branch.
• For the currently active HEAD, Git checks out a copy of the corresponding local branch.

Figure 12.3 illustrates the before and after state of a clone.

Notice that you start out with a remote repository containing a few commits and two separate branches: master and testing. From your point of view, these are the remote branches. When you do the clone, you get a copy of the remote repository cloned into a local repository. The existing branches in the remote each get their own remote tracking branches, namespaced with the name of the remote—that is, origin/master and origin/testing. For the active HEAD, a local branch, master, is created in the local repository and the contents are checked out into the working directory.

Remember that creating a branch is just creating a pointer in Git. A benefit of the remote tracking branches is to mark where the branches were located in the remote, so you can reference where they are pointing locally without having to maintain a persistent connection to the remote.

You can easily see the remote tracking branches after a clone by using the -r option on the git branch command.

$ git branch -r
  origin/HEAD -> origin/master
  origin/testing
  origin/master

Understanding Clone Paths

By default, a clone operation creates a local directory with the name of the repository. If there is a multi-level path, only the last part of the path is the actual repository. You can think of this like downloading a Zip file. The path in front of the Zip file is just the path; the last part is the actual Zip filename. And, when you download the Zip file, it is expanded into whatever structure is contained in the file. Likewise, the last piece of the repository path is the actual repository, and it is expanded into the .git structure and a checked-out version of the current HEAD. Figure 12.4 illustrates this idea.

In most cases, repositories avoid the issue of multiple levels in paths by hyphenating all of the levels together as one name. For example, top/mid/my.git becomes top-mid-my.git, and the directory that is created when this path is cloned becomes C: op-mid-my.

If you want to clone to a different destination directory (one that's different from the name of the repository), you can simply add that directory as the last argument to the clone command.

$ git clone [email protected]:top/mid/my.git project1

In this case, Git creates the local subdirectory as project1 and then clones the repository into that subdirectory instead of into my.

Viewing Information about Remote Branches

After cloning or establishing a connection to a remote branch and synching the content, there are several different ways to see information about remote branches.

First, you can view the list of remote branches with the -r option to git branch. The -r option here refers to remotes. Here's an example of how to use it:

$ git branch -r
  origin/HEAD -> origin/master
  origin/master
  origin/test
  origin/test2

In this case, you have three branches in the remote branch that you are now tracking in the local repository: master, test, and test2. Notice that this command also shows you where HEAD is pointing.

If you want to see the local branches as well as the remote ones, you can use the -a option (all) to the git branch command.

$ git branch -a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
  remotes/origin/test
  remotes/origin/test2

Notice that this list now includes your one local branch (master) along with the remote tracking branches.

For any of these options, you can add the -v (verbose) option to see which commit is current on the branch—where the tip of the branch points to. This example shows the short version of the commit's SHA1 and the commit message associated with that commit:

git branch -av
* master                539358f update file
  remotes/origin/HEAD   -> origin/master
  remotes/origin/master 539358f update file
  remotes/origin/test   8391dbd update
  remotes/origin/test2  8391dbd update

If you had used the -rv option instead, you would not have seen the first line with the local master branch.

There is one more variation you can use with the branch command to find out additional information: the -vv flag. (Note that this is two v's side by side, not one w.) The two v's tell Git to show extra information that is very useful—namely, any tracking connections between local branches and the remote tracking (upstream) branches.

$ git branch -vv
* master 539358f [origin/master] update file

This example is telling you that your local master branch is set up to track origin/master. You could also use an -a or -r option with the -vv option. However, because those options would only add the same remote tracking branch list, no additional information would be output.

One other way to get the list of branches indirectly is to use the show option for your remote.

$ git remote show origin
* remote origin
  Fetch URL: git@diyvb:repos/remote_demo
  Push  URL: git@diyvb:repos/remote_demo
  HEAD branch: master
  Remote branches:
    master     tracked
    test       tracked
    test2      tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (up to date)

At the bottom of the output, you can see that the local branch master is configured to work with the remote branch master.

Finally, if you take a look at the local config file, you can see where this configuration information is actually stored.

       $ cat .git/config
       [core]
             repositoryformatversion = 0
             filemode = true
             bare = false
             logallrefupdates = true
       [remote "origin"]
             url = git@diyvb:repos/remote_demo
             fetch = +refs/heads/*:refs/remotes/origin/*
       [branch "master"]
             remote = origin
             merge = refs/heads/master

Note the last section about branch master and the resemblance to the data from the git remote show output.

Now that you understand how to view the upstream and tracking branch information, let's look at how to set up the upstream and tracking relationship for branches that don't already have such a relationship.

$ git branch -av
* master                539358f update file
  remotes/origin/HEAD   -> origin/master
  remotes/origin/master 539358f update file
  remotes/origin/test   8391dbd update
  remotes/origin/test2  8391dbd update

$ git branch test

$ git branch -vv
* master 539358f [origin/master] update file
  test   539358f update file
$ git checkout test
Switched to branch 'test'

$ git checkout test2
Branch test2 set up to track remote branch test2 from origin.
Switched to a new branch 'test2'
$ git branch -vv
  master 539358f [origin/master] update file
  test   539358f update file
* test2  8391dbd [origin/test2] update

$ git branch test origin/test
Branch test set up to track remote branch test from origin.

$ git branch --track test origin/test
Branch test set up to track remote branch test from origin.
$ git branch -vv | grep test
  test   8391dbd [origin/test] update

$ git branch --no-track test origin/test
diyuser@diyvb:~/remote_demo4$ git branch -vv | grep test
  test   8391dbd update

$ git branch --set-upstream test2 origin/test2
The --set-upstream flag is deprecated and will be removed. Consider using --track or --set-upstream-to
Branch test2 set up to track remote branch test2 from origin. 

$ git branch test
$ git branch -vv | grep test
  test   539358f update file
$ git branch --set-upstream-to=origin/test test
Branch test set up to track remote branch test from origin.
$ git branch -vv | grep test
  test   539358f [origin/test: ahead 4, behind 1] update file

$ git branch test
$ git branch -vv | grep test
  test   539358f update file
$ git branch -u origin/test2 test
Branch test set up to track remote branch test2 from origin.
$ git branch -vv | grep test
  test   539358f [origin/test2: ahead 4, behind 1] update file

Push

As you saw in my earlier promotion model for Git, you can use the push command to update the remote repository. Basically, you can think of it as trying to take changes you've committed into your local repository and mirror them to the remote repository. As part of the same operation, push takes note of which commits the corresponding branches in the remote repository point to, and it updates the remote tracking branches in the local repository to point to the same mirrored versions there. The syntax for the push command is as follows:

git push [--all | --mirror | --tags] [--follow-tags] [--atomic] [-n | --dry-           run] [--receive-pack=<git-receive-pack>]
         [--repo=<repository>] [-f | --force] [-d | --delete] [--prune] [-v | --verbose]
          [-u | --set-upstream]
          [--[no-]signed|--sign=(true|false|if-asked)]
          [--force-with-lease[=<refname>[:<expect>]]]
          [--no-verify] [<repository> [<refspec>…]]

Let's look at an example of what a push operation would look like between a local repository and a remote repository. In this case, you'll mimic the idea of pushing to an empty remote repository from an existing local repository.

Figure 12.5 shows a local repository with three commits that have been made into it and an empty remote repository.

If you now do a push operation, Git takes your local changes and updates the remote repository with them (see Figure 12.6).

Git also creates a branch pointer in the local repository to indicate where the corresponding branch in the remote repository is pointing (see Figure 12.7). This creates the remote tracking branch that I have been talking about. This branch has the namespace of the remote reference (origin by default), so you can reference it as origin/master. You can now reference the origin/master branch as a representation of the master branch on the remote without being connected to the remote.

Seeing Status: Local versus Remote

Carrying this sequence one step further, suppose you now make a commit into the local repository that is not (yet) pushed over to the remote repository, as shown in Figure 12.8.

The interesting thing about this state is what a git status command would show.

$ git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)
nothing to commit, working directory clean

Notice the wording here. You can read the first sentence as: “Compared to the last time you checked in with the remote, your local branch has one newer commit on it.” This is because origin/master (the remote tracking branch) represents where master was the last time you communicated with the remote. To see the differences another way, you can use the git branch -av command.

$ git branch -av
* master                  253231d [ahead 1] <commit message>
  remotes/origin/master   bed5211 <previous commit message>

Notice the shorthand notation here, [ahead 1], which means the same thing as “master is ahead of origin/master by one commit.”

You can try one other variation by using the -vv option.

$ git branch -vv
* master   253231d [origin/master: ahead 1] <commit message>

$ git push <remote repository> <remote branch>

$ git push <remote repository> <local branch>:<remote branch>

$ git push

$ git push origin :

$ git push origin HEAD

$ git push origin branch1 branch2 branch3
$ git push origin lbranch1:rbranch1 lbranch2:rbranch2 lbranch3:rbranch3

$ git push --delete origin testing
- [deleted]         testing

$ git push origin :testing
- [deleted]         testing

$ git push
To C:/Program Files/Git/./calc2.git
 ! [rejected]        master -> master (fetch first)
error: failed to push some refs to 'C:/Program Files/Git/./calc2.git'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull …') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

$ git push -f origin master
Counting objects: 10, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (10/10), 856 bytes | 0 bytes/s, done.
Total 10 (delta 0), reused 0 (delta 0)
To C:/Program Files/Git/./calc2.git
 + 0caf33c…7f3fb23 master -> master (forced update)

$ git push origin +master
Counting objects: 14, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (14/14), 1.14 KiB | 0 bytes/s, done.
Total 14 (delta 0), reused 0 (delta 0)
To C:/Program Files/Git/./calc2.git
 + 7f3fb23…13f956a master -> master (forced update)

Fetch

Once a repository is cloned, the connection between the remote branches and remote tracking branches in the local repository is established. The fetch command allows you to get the latest updates to the remote tracking branches (as well as tags). The syntax is fairly straightforward.

git fetch [<options>] [<repository> [<refspec>…]]
git fetch [<options>] <group>
git fetch --multiple [<options>] [(<repository> | <group>)…]
git fetch --all [<options>]

To understand the basic mechanics of a fetch operation, take a look at Figure 12.9. This is a continuation of the set of commits you were using in the push discussion (shown in Figures 12.5 to 12.8).

On the left side of Figure 12.9 is your starting point. You've pushed the most recent commit you made locally over to the remote. However, someone else has also made a change and pushed it into the remote repository. Thus, you have an additional commit in the remote repository that you don't have in your local repository.

If you were to do a git status command at this point, you would get the following output:

$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

This may seem confusing at first because I just said that there was newer content in the remote repository that you don't have in your local repository. However, remember that you're using the remote tracking branch (origin/master) as your bookmark of where the corresponding branch on the remote is. And the remote tracking branch is only updated when you check in with the remote. So, based on the last time you checked in with the remote, both the remote master branch and your local master branch were pointing at the same commit.

Now, if you look at the right side of Figure 12.9, you see a fetch operation being performed. The fetch operation communicates with the remote repository to see if there are any updates there that it needs to reflect in the corresponding remote tracking branch. There are: there is a new commit on master in the remote repository, as I previously noted. So, Git updates the repository with the new commit and (only) moves the remote tracking branch (origin/master) to point to the new commit. Notice that your local branch, master, is not updated. Fetch only updates the contents of the repository and remote tracking branches.

After the fetch operation, if you check the status, you see something like this:

$ git status
On branch master
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)
nothing to commit, working directory clean

This tells you that your current branch master is one commit behind where the remote tracking branch is. (Another way to say this is that the remote tracking branch, and thus the remote, has one newer commit that your local branch is not yet aware of.) Notice that the output also tells you that your local branch can be fast-forwarded. I'll discuss what that means shortly.

Fetch Options

As with the push command, there are a few options you should know about for the fetch command.

depth

The depth option limits fetching to the number of commits specified as an argument to the option. For example, --depth=1 only fetches down the latest changes (latest commit).

force

Like push, fetch also has a force option. If you attempt to fetch from a branch on the remote repository to a remote tracking branch in the local repository and Git can't do a fast-forward merge, then it refuses to do the update. However, you can use the --force (-f) option to force Git to do the update. Of course, you should only do this if you understand the consequences and are sure you want to do it.

To illustrate this, I'll use an example of trying to fetch updates from one branch into a completely different branch. Attempting this when Git can't do a fast-forward merge between the two branches results in an error message like the following one:

$ git fetch origin master:ui
! [rejected]        master     -> ui  (non-fast-forward)

You can use the -f option to force the update to happen.

$ git fetch -f origin master:ui
+ fbcf72c…8a3f48b master     -> ui  (forced update) 

As an optional method of forcing the update, you can supply a plus (+) sign in front of the branches.

$ git fetch origin +master:ui
+ abaf124…8a3f48b master     -> ui  (forced update)

The -f (or --force) option is an option to fetch, and it applies to all branches passed to the command. You can use the plus (+) sign to specify forcing any particular branch if you don't want to do all of them.

Synchronizing Local Branches after a Fetch

Recall that in an earlier example, after fetching the latest updates, the status said that your local branch was one commit behind the remote tracking branch and could be “fast-forwarded.”

This means that a fast-forward merge can be done to bring the local branch up-to-date with the latest changes to the remote tracking branch (and thus the remote branch). The left side of Figure 12.10 shows the local repository after the fetch but before the merge. Note that the master branch is one commit behind the remote tracking branch.

You can then do the merge.

$ git merge origin/master
Updating 253231d..ecf2390
Fast-forward
 …

You end up with the results on the right side of Figure 12.10. Notice that the master branch has been fast-forwarded to the same commit as the remote-tracking branch. Also, as I discuss in Chapter 9, the merge occurs in the working directory as well as the local repository. This is shown in Figure 12.10 by the change in contents in the working directory before and after the merge.

This is how you synchronize a local branch after a fetch updates. After the merge, your local repository, local branch, and files in the working directory all have the latest updates from the remote.

Pull

If you understand fetch and merge, then you understand the pull command. In its default form, the git pull command is essentially a fetch followed by a merge. Figure 12.11 represents a before-and-after view that is similar to the other commands.

The syntax for the pull command is also fairly straightforward:

git pull [options] [<repository> [<refspec>…]]

Let's look at a few of the options for the pull command.

$ git pull origin features:docs
From https://github.com/brentlaster/calc2
 ! [rejected]        features   -> docs  (non-fast-forward)

$ git pull -f origin features:docs
From https://github.com/brentlaster/calc2
 + abaf124…b372aa6 features   -> docs  (forced update)

$ git pull origin +features:docs
From https://github.com/brentlaster/calc2
 + abaf124…b372aa6 features   -> docs  (forced update)

About Connected Lab 8: Setting Up a GitHub Account and Cloning a Repository

This lab is intended to give you some practice in working with remotes. To do this, you'll set up a GitHub repository and see how some of the basic concepts such as forking, cloning, and adding remote references work in practice.