Teach "stg import" to import patch series from tar archives

[stgit] / Documentation / tutorial.txt

StGit tutorial
##############

StGit is a command-line application that provides functionality
similar to htmllink:http://savannah.nongnu.org/projects/quilt/[Quilt]
(i.e. pushing/popping patches to/from a stack), but using Git instead
of +diff+ and +patch+. StGit stores its patches in a Git repository as
normal Git commits, and provides a number of commands to manipulate
them in various ways.

This tutorial assumes you are already familiar with the basics of Git
(for example, branches, commits, and conflicts). For more information
on Git, see manlink:git[1] or htmllink:http://git.or.cz/[the Git home
page].


Help
====

For a full list of StGit commands:

  $ stg help

For quick help on individual subcommands:

  $ stg help <cmd>

For more extensive help on a subcommand:

  $ man stg-<cmd>

(The documentation is also available in htmllink:stg.html[HTML
format].)


Getting started
===============

StGit is not a stand-alone program -- it operates on a Git repository
that you have already created, using +git init+ or +git clone+. So get
one of those; if you don't have one at hand, try for example

  $ git clone http://homepage.ntlworld.com/cmarinas/stgit.git
  $ cd stgit

Before you can create StGit patches, you have to run stglink:init[]:

  $ stg init

This initializes the StGit metadata for the current branch. (So if you
want to have StGit patches in another branch too, you need to run +stg
init+ again in that branch.)

NOTE: As a shortcut, stglink:clone[] will run +git clone+ followed by
+stg init+ for you.


Creating a patch
----------------

Now we're ready to create our first patch:

  $ stg new my-first-patch

This will create a patch called +my-first-patch+, and open an editor
to let you edit the patch's commit message. (If you don't give a name
on the command line, StGit will make one up based on the first line of
the commit message.) This patch is empty, as stglink:show[] will tell
you:

  $ stg show

But it won't stay that way for long! Open one of the files in your
favorite text editor, change something, and save. You now have some
local changes in your tree:

  $ stg status
  M stgit/main.py

Then stgsublink:refresh[] the patch:

  $ stg refresh

And voilà -- the patch is no longer empty:

  $ stg show
  commit 3de32068c600d40d8af2a9cf1f1c762570ae9610
  Author: Audrey U. Thor <author@example.com>
  Date:   Sat Oct 4 16:10:54 2008 +0200

      Tell the world that I've made a patch

  diff --git a/stgit/main.py b/stgit/main.py
  index e324179..6398958 100644
  --- a/stgit/main.py
  +++ b/stgit/main.py
  @@ -171,6 +171,7 @@ def _main():
       sys.exit(ret or utils.STGIT_SUCCESS)

   def main():
  +    print 'My first patch!'
       try:
           _main()
       finally:

(I'm assuming you're already familiar with
htmllink:http://en.wikipedia.org/wiki/Diff#Unified_format[unified
diff] patches like this from Git, but it's really quite simple; in
this example, I've added the +$$print 'My first patch!'$$+ line to the
file +stgit/main.py+, at around line 171.)

Since the patch is also a regular Git commit, you can also look at it
with regular Git tools such as manlink:gitk[].

Creating another patch
----------------------

We want to make another improvement, so let's create a new patch for
it:

  $ echo 'Audrey U. Thor' > AUTHORS
  $ stg new credit --message 'Give me some credit'
  $ stg refresh

Note that we can give the commit message on the command line, and that
it doesn't matter whether we run stglink:new[] before or after we edit
the files.

So now we have two patches:

  $ stg series --description
  + my-first-patch # This is my first patch
  > credit         # Give me some credit

stglink:series[] lists the patches from bottom to top; +$$+$$+ means
that a patch is 'applied', and +>+ that it is the 'current', or
topmost, patch.

If we want to make further changes to the topmost patch, we just edit
the files and run +stg refresh+. But what if we wanted to change
+my-first-patch+? The simplest way is to stgsublink:pop[] the +credit+
patch, so that +my-first-patch+ becomes topmost again:

  $ stg pop credit
  Checking for changes in the working directory ... done
  Popping patch "credit" ... done
  Now at patch "my-first-patch"
  $ stg series --description
  > my-first-patch # This is my first patch
  - credit         # Give me some credit

stglink:series[] now shows that +my-first-patch+ is topmost again,
which means that stglink:refresh[] will update it with any changes we
make.

The minus sign says that +credit+ is 'unapplied' -- this means that
it's been temporarily put aside. If you look at the +AUTHORS+ file,
you'll see that our change to it is gone; and tools such as
manlink:gitk[] will not show it, because it's been edited out of the
Git history. But it's just one stglink:push[] command away from being
restored:

  $ stg push credit
  Checking for changes in the working directory ... done
  Fast-forwarded patch "credit"
  Now at patch "credit"

NOTE: You can omit the patch name argument to stglink:push[] and
stglink:pop[]. If you do, you will push the next unapplied patch, and
pop the topmost patch, respectively.

NOTE: There are at least two more ways to update a non-topmost patch.
One is to use stglink:refresh[] with the +$$--patch$$+ flag, the other
to create a new patch for the update and then merge it into the other
patch with stglink:coalesce[].


Keeping commit messages up to date
----------------------------------

Since StGit is all about creating readable Git history (or a readable
patch series, which is essentially the same thing), one thing you'll
want to pay attention to is the commit messages of your patches.
stglink:new[] asks you for a commit message when you create a new
patch, but as time goes by and you refresh the patch again and again,
chances are that the original commit message isn't quite correct
anymore. Fortunately, editing the commit message is very easy:

  $ stg edit <patch-name>

In addition to stglink:edit[], you can also give the +$$--edit$$+ flag
to stglink:refresh[] -- that way, you get to change the commit message
and update the patch at the same time. Use whichever feels most
natural to you.

NOTE: stglink:edit[] has a +$$--diff$$+ flag, which gives you the diff
text and not just the commit message in your editor. Be aware, though,
that if you change the diff so that it no longer applies, the edit
will be saved to a file instead of being carried out. If you're not
comfortable editing diffs, just treat +$$--diff$$+ as a way to get to
'see' the diff while you edit the commit message.

If the patch changes considerably, it might even deserve a new name.
stglink:rename[] is your friend there.


Conflicts
---------

Normally, when you pop a patch, change something, and then later push
it again, StGit sorts out everything for you automatically. For
example, let's create two patches that modify different files:

  $ stg clone http://homepage.ntlworld.com/cmarinas/stgit.git stgit
  $ cd stgit
  $ stg new first --message 'First patch'
  $ echo '- Do something' >> TODO
  $ stg refresh
  $ stg new second --message 'Second patch'
  $ echo '- Install something' >> INSTALL
  $ stg refresh

then pop them both:

  $ stg pop --all

and then push them in the opposite order:

  $ stg push second first
  $ stg series
  + second
  > first

StGit had no problems reordering these patches for us, since they
didn't touch the same file. But it would have worked just fine even if
they had touched the same file, as long as they didn't change the same
part of the file. But what if they did? Let's find out.

  $ stg pop
  Checking for changes in the working directory ... done
  Popping patch "first" ... done
  Now at patch "second"
  $ echo '- Do something else' >> TODO
  $ stg refresh

Now, both patches add a new line at the end of +TODO+. So what happens
when we try to have them both applied?

  $ stg push
  Pushing patch "first" ...
    CONFLICT (content): Merge conflict in TODO
    Error: The merge failed during "push".
           Revert the operation with "stg undo".
    stg push: 1 conflict(s)

StGit is telling us that it couldn't figure out how to push +first+ on
top of +second+, now that they both modify +TODO+. We can take a look
at the situation with stglink:status[]:

  $ stg status
  ? TODO.ancestor
  ? TODO.current
  ? TODO.patched
  C TODO

As we were told by stglink:push[], the conflict is in the file +TODO+.
(If the patch was bigger and touched multiple files, they would all be
listed here; prefixed with +C+ if they had conflicts, and +M+ if StGit
managed to automatically resolve everything in the file.)

NOTE: +TODO.ancestor+, +TODO.current+, and +TODO.patched+ are the
three versions of the file that StGit tried to merge. The +.current+
file is the version before the patch was applied, +.patched+ is the
version in the patch we tried to push, and +.ancestor+ the version
that contains neither of the added lines.

At this point, we have two options:

  1. Undo the failed merge with stglink:undo[]. (Remember to use the
     +$$--hard$$+ flag, since the unresolved conflict means the
     worktree is not clean.)

  2. Manually resolve the conflict.

To resolve the conflict, open +TODO+ in your favorite editor. It ends
like this:

----------------------------------------------------------------------
- numeric shortcuts for naming patches near top (eg. +1, -2)
- (config?) parameter for number of patches included by "series -s"
<<<<<<< current:TODO
- Do something else
=======
- Do something
>>>>>>> patched:TODO
----------------------------------------------------------------------

The 'conflict markers' +<<<<<<<+, +=======+, and +>>>>>>>+ indicate
which lines were already there (+current+) and which were added by the
patch (+patched+). Edit the file so that it looks like it should; in
this case, we want something like this:

----------------------------------------------------------------------
- numeric shortcuts for naming patches near top (eg. +1, -2)
- (config?) parameter for number of patches included by "series -s"
- Do something
- Do something else
----------------------------------------------------------------------

Note that ``looks like it should'' includes removing the conflict
markers.

Now that we've resolved the conflict, we just need to tell StGit about
it:

  $ stg resolved TODO
  $ stg status
  M TODO

+TODO+ is listed as being modified, not in conflict. And we know from
before how to deal with modified files:

  $ stg refresh

The conflict is now resolved. We can see that +first+ now looks a
little different; it no longer adds a line at the end of the file:

  $ stg show
  commit 8e3ae5f6fa6e9a5f831353524da5e0b91727338e
  Author: Audrey U. Thor <author@example.com>
  Date:   Sun Oct 5 14:43:42 2008 +0200

      First patch

  diff --git a/TODO b/TODO
  index 812d236..4ef3841 100644
  --- a/TODO
  +++ b/TODO
  @@ -24,4 +24,5 @@ The future, when time allows or if someone else does them:
     they have scripts for moving the changes in one to the others)
   - numeric shortcuts for naming patches near top (eg. +1, -2)
   - (config?) parameter for number of patches included by "series -s"
  +- Do something
   - Do something else


Workflow: Development branch
============================

One common use of StGit is to ``polish'' a Git branch before you
publish it for others to see. Such history falsification can often be
a 'good' thing -- when you (or someone else) needs to look at what you
did six months later, you are not really interested in all the false
starts and the steps needed to corect them. What you want is the final
solution, presented in a way that makes it easy to read and
understand.

Of course, there are limits. Editing the last few days' worth of
history is probably a good idea; editing the last few months' probably
isn't. A rule of thumb might be to not mess with history old enough
that you don't remember the details anymore. And rewriting history
that you have published for others to see (and base their own work on)
usually just makes everyone more confused, not less.

So, let's take a concrete example. Say that you're hacking on StGit,
and have made several Git commits as your work progressed, with commit
messages such as ``Improve the snarfle cache'', ``Remove debug
printout'', ``New snarfle cache test'', ``Oops, spell function name
correctly'', ``Fix documentation error'', and ``More snarfle cache''.

Now, this is the actual history, but for obvious reasons, this isn't
the kind of history you'd ideally want to find when you six months
from now try to figure out exactly where that elusive snarfle cache
bug was introduced. So let's turn this into the history we can be
proud of. The first step is to make StGit patches out of all those Git
commits:

  $ stg uncommit --number 6
  Uncommitting 6 patches ...
    Now at patch "more-snarfle-cache"
  done
  $ stg series --description
  + improve-the-snarfle-cache      # Improve the snarfle cache
  + remove-debug-printout          # Remove debug printout
  + new-snarfle-cache-test         # New snarfle cache test
  + oops-spell-function-name-corre # Oops, spell function name correctly
  + fix-documentation-error        # Fix documentation error
  > more-snarfle-cache             # More snarfle cache

As you can see, stglink:uncommit[] adds StGit metadata to the last few
Git commits, turning them into StGit patches so that we can do stuff
with them.

NOTE: With the +$$--number$$+ flag, stglink:uncommit[] uncommits that
many commits and generates names for them based on their commit
messages. If you like, you can instead list the patch names you want
on the command line.

At this point, there are a number of things we could do:

  * Continue developing, and take advantage of e.g. stglink:goto[] or
    +stg refresh $$--patch$$+ to stick updates in the right patch to
    begin with.

  * Use e.g. stglink:float[], stglink:sink[], stglink:push[], and
    stglink:pop[] to reorder patches.

  * Use stglink:coalesce[] to merge two or more patches into one.
    stgsublink:coalesce[] pushes and pops so that the patches to be
    merged are consecutive and unrelated patches aren't in the way,
    then makes one big patch out of the patches to be merged, and
    finally pushes the other patches back.
+
Of course, as always when there is pushing involved, there is the
possibility of conflicts. If a push results in a conflict, the
operation will be halted, and we'll be given the option of either
resolving the conflict or undoing.

Once we feel that the history is as good as it's going to get, we can
remove the StGit metadata, turning the patches back into regular Git
commits again:

  $ stg commit --all

TIP: stglink:commit[] can also commit specific patches (named on the
command line), leaving the rest alone. This can be used to retire
patches as they mature, while keeping the newer and more volatile
patches as patches.


Workflow: Tracking branch
=========================

In the 'Development branch' workflow described above, we didn't have
to worry about other people; we're working on our branch, they are
presumably working on theirs, and when the time comes and we're ready
to publish our branch, we'll probably end up merging our branch with
those other peoples'. That's how Git is designed to work.

Or rather, one of the ways Git is designed to work. An alternative,
popular in e.g. the Linux kernel community (for which Git was
originally created), is that contributors send their patches by e-mail
to a mailing list. Others read the patches, try them out, and provide
feedback; often, the patch author is asked to send a new and improved
version of the patches. Once the project maintainer is satisfied that
the patches are good, she'll 'apply' them to a branch and publish it.

StGit is ideally suited for the process of creating patches, mailing
them out for review, revising them, mailing them off again, and
eventually getting them accepted.


Getting patches upstream
------------------------

We've already covered how to clone a Git repository and start writing
patches. As for the next step, there are two commands you might use to
get patches out of StGit: stglink:mail[] and stglink:export[].
stglink:export[] will export your patches to a filesystem directory as
one text file per patch, which can be useful if you are going to send
the patches by something other than e-mail. Most of the time, though,
stglink:mail[] is what you want.

NOTE: Git comes with tools for sending commits via e-mail. Since StGit
patches are Git commits, you can use the Git tools if you like them
better for some reason.

NOTE: For exporting single patches -- as opposed to a whole bunch of
them -- you could also use stglink:show[] or stglink:diff[].

Mailing a patch is as easy as this:

  $ stg mail --to recipient@example.com <patches>

You can list one or more patches, or ranges of patches. Each patch
will be sent as a separate mail, with the first line of the commit
message as subject line. Try mailing patches to yourself to see what
the result looks like.

NOTE: stglink:mail[] uses +sendmail+ on your computer to send the
mails. If you don't have +sendmail+ properly set up, you can instruct
it to use any SMTP server with the +$$--smtp-server$$+ flag.

There are many command-line options to control exactly how mails are
sent, as well as a message template you can modify if you want. The
man page has all the details; I'll just mention two more here.

+$$--edit-cover$$+ will open an editor and let you write an
introductory message; all the patch mails will then be sent as replies
to this 'cover message'. This is usually a good idea if you send more
than one patch, so that reviewers can get a quick overview of the
patches you sent.

+$$--edit-patches$$+ will let you edit each patch before it is sent.
You can change anything, but note that you are only editing the
outgoing mail, not the patch itself; if you want to make changes to
the patch, you probably want to use the regular StGit commands to do
so. What this 'is' useful for, though, is to add notes for the patch
recipients:

----------------------------------------------------------------------
From: Audrey U. Thor <author@example.com>
Subject: [PATCH] First line of the commit message

The rest of the commit message

---

Everything after the line with the three dashes and before the diff is
just a comment, and not part of the commit message. If there's
anything you want the patch recipients to see, but that shouldn't be
recorded in the history if the patch is accepted, write it here.

 stgit/main.py |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/stgit/main.py b/stgit/main.py
index e324179..6398958 100644
--- a/stgit/main.py
+++ b/stgit/main.py
@@ -171,6 +171,7 @@ def _main():
     sys.exit(ret or utils.STGIT_SUCCESS)

 def main():
+    print 'My first patch!'
     try:
         _main()
     finally:
----------------------------------------------------------------------


Rebasing a patch series
-----------------------

While you are busy writing, submitting, and revising your patch
series, other people will be doing the same thing. As a result, even
though you started writing your patches on top of what was the latest
history at the time, your stack base will grow ever more out of date.

When you clone a repository,

  $ stg clone http://homepage.ntlworld.com/cmarinas/stgit.git stgit

you initially get one local branch, +master+. You also get a number of
'remote' branches, one for each branch in the repository you cloned.
In the case of the StGit repository, these are
+remotes/origin/stable+, +remotes/origin/master+, and
+remotes/origin/proposed+. +remotes+ means that it's not a local
branch, just a snapshot of a branch in another repository; and
+origin+ is the default name for the first remote repository (you can
set up more; see the man page for +git remote+).

Right after cloning, +master+ and +remotes/origin/master+ point at the
same commit. When you start writing patches, +master+ will advance,
and always point at the current topmost patch, but
+remotes/origin/master+ will stay the same because it represents the
master branch in the repository you cloned from -- your 'upstream'
repository.

Unless you are the only one working on the project, however, the
upstream repository will not stay the same forever. New commits will
be added to its branches; to update your clone, run

  $ git remote update

This will update all your remote branches, but won't touch your local
branches. To get the latest changes into your local +master+ branch,
use stglink:rebase[]:

  $ stg rebase remotes/origin/master

This command will do three things:

  1. Pop all patches, so that your local branch (+master+, in this
     example) points at the stack base. This is the same commit that
     +remotes/origin/master+ pointed at at the time you started
     writing your patches.

  2. Set the stack base to the given commit (the current, updated
     value of +remotes/origin/master+).

  3. Push the patches that were popped in the first step.

The end result is that your patches are now applied on top of the
latest version of +remotes/origin/master+.

The primary reason for rebasing is to reduce the amount of conflicts
between your work and others'. If one of your patches changes the same
part of the same file as a patch someone else has written, you will
get a conflict when you run stglink:rebase[] the next time after the
other person's patch has been accepted upstream. It is almost always
less work to rebase often and resolve these one at a time, rather than
a whole lot at once. After all, you have to rebase eventually; if you
mail out patches that are based on an outdated branch, everyone who
tries to apply them has to resolve the conflicts instead. There are
more effective ways to get popular.


When your patches are accepted
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If and when some or all of your patches are accepted upstream, you
update and rebase just like usual -- but be sure to use the
+$$--merged$$+ flag to stglink:rebase[]:

  $ git remote update
  $ stg rebase --merged remotes/origin/master

This flag makes the rebase operation better at detecting that your
patches have been merged, at some cost in performance.

The patches that had been merged will still be present in your patch
stack after the rebase, but they will be empty, since the change they
added is now already present in the stack base. Run stglink:clean[] to
get rid of such empty patches if you don't want them hanging around:

  $ stg clean


Importing patches
-----------------

TODO:: import, ...


Other stuff that needs to be placed somewhere
=============================================


Undo
----

TODO:: undo, redo, log, reset


Interoperating with Git
-----------------------

TODO::

* git commit + repair

* git reset HEAD~n + repair

* don't do git rebase or git merge, because it won't work


Patch stuff
-----------

TODO:: This section needs revising. I've only fixed the formatting.
Most of it should go under "Workflow: Tracking branch"

As mentioned in the introduction, StGit stores modifications to your
working tree in the form of Git commits. This means if you want to
apply your changes to a tree not managed by Git, or send your changes
to someone else in e-mail, you need to convert your StGit patches into
normal textual diffs that can be applied with the GNU patch command.
stglink:diff[] is a powerful way to generate and view textual diffs of
patches managed by StGit.

To view a diff of the topmost patch:

  $ stg diff -r /

Observe that this does not show any changes in the working directory
that have not been saved by a stgsublink:refresh[]. To view just the
changes you've made since the last refresh, use:

  $ stg diff -r /top

If you want to see the changes made by the patch combined with any
unsaved changes in the working directory, try:

  $ stg diff -r /bottom

You can also show the changes to any patch in your stack with:

  $ stg diff -r <patch>/

Use this command to view all the changes in your stack up through the
current patch:

  $ stg diff -r base

stglink:diff[] supports a number of other features that are very
useful. Be sure to take a look at the help information for this
command. To convert your StGit patches into patch files:

  $ stg export [--range=[<patch1>[:<patch2>]]] [<dir-name>]

stglink:export[] supports options to automatically number the patches
(+-n+) or add the +.diff+ extension (+-d+). If you don't tell
stgsublink:export[] where to put the patches, it will create directory
named +patch-<branchname>+ in your current directory, and store the
patches there. To e-mail a patch or range of patches:

  $ stg mail [--to=...] (--all | --range=[<patch1>[:<patch2>]] | <patch>)

stglink:mail[] has a lot of options, so read the output of +stg mail
-h+ for more information.

You can also import an existing GNU diff patch file as a new StGit
patch with a single command. stglink:import[] will automatically parse
through the patch file and extract a patch description. Use:

  $ stg import [<file>]

This is the equivalent of

  $ stg new
  $ patch -i <file>
  $ stg refresh -e

Sometimes the patch file won't apply cleanly. In that case,
stglink:import[] will leave you with an empty StGit patch, to which
you then apply the patch file by hand using "patch -i" and your
favorite editor.

To merge a GNU diff file (defaulting to the standard input) into the
topmost patch:

  $ stg fold [<file>]

This command supports a +$$--threeway$$+ option which applies the
patch onto the bottom of the topmost one and performs a three-way
merge.


Templates
---------

TODO:: This section needs revising. I've only fixed the formatting.

stglink:export[] and stglink:mail[] use templates for generating the
patch files or e-mails. The default templates are installed under
+<prefix>/share/stgit/templates/+ and, combined with the extra options
available for these commands, should be enough for most users. The
template format uses the standard Python string formatting rules. The
variables available are listed in the the manual pages for each
command. stglink:mail[] can also send an initial 'cover' e-mail for
which there is no default template. The
+<prefix>/share/stgit/examples/firstmail.tmpl+ file can be used as an
example. A default description for new patches can be defined in the
+.git/ patchdescr.tmpl+ file. This is useful for things like
signed-off-by lines.