2009-10-09

OpenGrok on Ubuntu w/ Tomcat6

A friend of mine turned me onto OpenGrok a couple of years ago, so when we migrated from CVS to SVN at work I set it up on one of our dev webapp servers to give my team a way to search the repository. The thing I like the most about it is that it does syntax highlighting of your source so you can easily browse without checking a project out into an IDE. There are a lot more advanced features, but that's the one I appreciate the most since I do a lot of browsing of legacy code.
Before we get into OpenGrok setup, use your favorite package manager to install the following; I'm assuming you already have an SVN repository with source.
  • Java >= 1.5; package sun-java6-jdk
  • Exuberant Ctags; package exuberant-ctags
  • Tomcat 6; package tomcat6
  • Tomcat 6 Manager; package tomcat6-admin (optional)

Tomcat tweaks

The Tomcat Manager webapp is a very useful tool that provides server & webapp status as well as a few utilities. In order to use it however, you need to setup a login, which is done in /etc/tomcat6/tomcat-users.xml. Add the following line, using whatever username and password you choose; it doesn't have to match your Ubuntu user.
<user username="username" password="password" roles="manager" />
Restart tomcat.
sudo /etc/init.d/tomcat6 restart
It's setup to run on port 8080 by default, so to get to the manager webapp, go to http://localhost:8080/manager/html and login with the credentials you just created.
Tomcat ships with a security policy enabled which provides the rest of your system a level of protection from webapps, however configuration is apparently pretty tricky. On my server and in this tutorial I've decided to keep it enabled, but if you trust tomcat and any webapp you might deploy therein, edit /etc/init.d/tomcat6 as follows to disable it.
TOMCAT6_SECURITY=no
If you do, you can skip the policy modifications herein, but I'm not liable for any ill effects that result from your decision.
The security policy in Ubuntu's tomcat6 package seems to be configured such that it is unable to write to its log (unintentionally I'm sure, though the maintainer should have tested this), so to correct that we can edit /etc/tomcat6/policy.d/03catalina.policy to match the tomcat6 SecurityManager HOWTO:
// ========== CATALINA CODE PERMISSIONS =======================================

// These permissions apply to the daemon code
grant codeBase "file:${catalina.home}/bin/commons-daemon.jar" {
        permission java.security.AllPermission;
};

// These permissions apply to the logging API
grant codeBase "file:${catalina.home}/bin/tomcat-juli.jar" {
    permission java.security.AllPermission;
};

// These permissions apply to the server startup code
grant codeBase "file:${catalina.home}/bin/bootstrap.jar" {
        permission java.security.AllPermission;
};

// These permissions apply to the servlet API classes
// and those that are shared across all class loaders
// located in the "lib" directory
grant codeBase "file:${catalina.home}/lib/-" {
        permission java.security.AllPermission;
};

Now on to OpenGrok

  1. Download the OpenGrok tarball.
    Go to OpenGrok's main page for up-to-date links.
    cd ~/src/
    wget http://opensolaris.org/os/project/opengrok/files/opengrok-0.7.tar.gz
    tar -xvzf opengrok-0.7.tar.gz
    cd opengrok-0.7
  2. Create your OpenGrok directory tree.
    The EXAMPLES.txt file that comes with OpenGrok is very useful, but I think it's silly to make a new directory at the filesystem root for OpenGrok.
    sudo mkdir -p /var/lib/opengrok/bin
    sudo mkdir -p /var/lib/opengrok/data
    sudo mkdir -p /var/lib/opengrok/src
    sudo mkdir /var/log/opengrok/
  3. Copy the index generation files.
    cp -r run.sh OpenGrok opengrok.jar lib /var/lib/opengrok/bin/
  4. Checkout your repo(s) into OpenGrok's source tree.
    I have many separate repositories, but OpenGrok handles this well.
    cd /var/lib/opengrok/src/
    svn checkout <repo1_URL>
    svn checkout <repo2_URL>
  5. Generate the OpenGrok config file, index.
    This command will write the config file for the webapp as well as create the source index for the webapp. If you have a single repository, remove the -P.
    sudo java -jar /var/lib/opengrok/bin/opengrok.jar -W /var/lib/opengrok/configuration.xml -c /usr/bin/ctags -P -s /var/lib/opengrok/src/ -d /var/lib/opengrok/data -w opengrok -L polished
    You'll see output (courtesy of -v) as OpenGrok analyzes your source tree. Run java -jar /var/lib/opengrok/bin/opengrok.jar for complete usage.
  6. Setup the webapp.
    We need to edit the webapp's web.xml so that it reads the configuration file we generated in the last step.
    unzip source.war -d source
    cd source
    vi WEB-INF/web.xml
    Change the CONFIGURATION context-param to point to our configuration file, but be sure to leave the ConfigAddress alone since this is required for source history and annotation to work.
      <context-param>
        <param-name>CONFIGURATION</param-name>
        <param-value>/var/lib/opengrok/configuration.xml</param-value>
        <description>Full path to the configuration file where OpenGrok can read it's configuration</description>
      </context-param>
    Now we update the web.xml in the war and deploy the webapp as opengrok.war. This matches the context we specified when generating the index.
    zip -u ../source.war WEB-INF/web.xml
    sudo cp ../source.war /var/lib/tomcat6/webapps/opengrok.war
  7. Grant OpenGrok access to the filesystem.
    Add the following to /etc/tomcat6/policy.d/50local.policy.
    grant codeBase "file:${catalina.base}/webapps/opengrok/-" {
         permission java.security.AllPermission;
    };
    
    grant codeBase "file:${catalina.base}/webapps/opengrok/WEB-INF/lib/-" {
         permission java.security.AllPermission;
    };
  8. Test the webapp
    Visit http://localhost:8080/opengrok/. From here there are several different search method; hit the 'browsable and searchable source tree' link to browse your repository. At this point you could leave it as-is, but it won't recognize future revisions, so we need to automate index updates.
  9. Configure the index generation script.
    Edit /var/lib/opengrok/bin/run.sh and make the following changes. Since we have all our settings in configuration.xml for the webapp, we'll use that instead of setting everything again here.
    #java ${JAVA_OPTS} ${PROPERTIES} ${LOGGER} -jar ${PROGDIR}/opengrok.jar ${SCAN_FOR_REPOS} -c ${EXUB_CTAGS} -s ${SRC_ROOT} -d ${DATA_ROOT}
    
    svn update ${PROGDIR}/../src/*
    
    java ${JAVA_OPTS} ${PROPERTIES} ${LOGGER} -jar ${PROGDIR}/opengrok.jar ${SCAN_FOR_REPOS} -R ${PROGDIR}/../configuration.xml
    
    # OPTIONAL
    #java ${LOGGER} -classpath ${PROGDIR}/opengrok.jar org.opensolaris.opengrok.web.EftarFile ${PATH_DESC} ${DATA_ROOT}/index/dtags.eftar
  10. Setup the index generation script in cron.
    Edit root's crontab:
    sudo crontab -e
    Add the following line to run the index generation at 5 minutes past every hour, every day and write output to a log. Index generation is incremental, so the runtime is much shorter than the initial run.
    5 * * * * /var/lib/opengrok/bin/run.sh >> /var/log/opengrok/index.log 2>&1
    
Let me know if you run into any problems.

20 comments:

  1. Thanks much for the guide. I see that not all files are listed when viewed from IE. With Firefox it works great. I don't have any issue using Firefox, but would be nice if issues IE are resolved. Do you know what could be the issue.

    Thanks again for the instructions

    ReplyDelete
  2. I don't think I've ever tried it from IE; do you see the HTML for the missing files if you view the source?

    ReplyDelete
  3. Surprisingly now all the files are listed. Prior to that if search for any function name that was in one of the invisible files then the file would be seen. Anyway now all the files are listed. However the mouse's roll or keyboards arrows/page up and page down buttons are not working on IE. I need to press the mouse key and drag the scroll bar down to see the full page.

    ReplyDelete
  4. Thanks for the post -- I found this very helpful! I'm still trying to get the history search working (with Mercurial)...any tips would be appreciated!

    ReplyDelete
  5. I've never setup OpenGrok for Mercurial, sorry. If the repository isn't local, make sure to use the "-r on" option.

    ReplyDelete
  6. Thanks for the post...
    But, I'm trying to setup OpenGrok for sources of three different products, and their source roots are not under common root directory. Besides, I want to narrow down the searches of different products's source to just that source root.

    So, I thought setting up different contexts and setting up different .war files to deploy on Tomcat would do the trick. But, unfortunately, for some source code elements, it just defaults to /source/s=? and it doesn't find the element. I know the element is defined in the source code...

    Any extra docs you can put out for this kind set up would be great.

    ReplyDelete
  7. @Mahendra - each directory within your OpenGrok source directory will be treated as a separate project when you use the -P option (as above) during index generation, so you may want to export or checkout each of the products you want to browse/search to the OpenGrok source directory.
    I do the same when I don't want to export an entire repository - I just export the projects that I care about, then generate the index with -P.

    A couple of other points:
    You can always restrict your search to the current directory tree while browsing by using the checkbox next to the search field at the top of the page.
    Also, you can define the webapp context used in source code links by using the -w option during index generation. This is necessary to make links work if you change the war name from the default, as I do in step 5 above.

    ReplyDelete
  8. Thank you again... I could define different webapp context using -w option...
    One more thing I'm trying is to have multibyte characters (japanese) in opengrok... I don't want those in index but should be readable at-least in web page (xref). Any help on this?

    ReplyDelete
  9. @Mahendra - I don't have any experience with other languages in OpenGrok, sorry.

    ReplyDelete
  10. Excellent Guide. Thank you very much Mike.

    ReplyDelete
  11. Mike, excellent write up for installing opengrok in ubuntu.

    I have a problem in creating indexes as mentioned in step 5. I don't work on Java nor i know meaning of exceptions in java. Can you help me out with below error when I execute step 5.

    -------------------------------
    Feb 20, 2011 8:05:32 PM org.opensolaris.opengrok.index.Indexer doIndexerExecution
    INFO: Starting indexing
    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/lucene/store/LockFactory
    at org.opensolaris.opengrok.index.Indexer.doIndexerExecution(Indexer.java:607)
    at org.opensolaris.opengrok.index.Indexer.main(Indexer.java:460)
    Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.LockFactory
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    ... 2 more
    -------------------------------------

    My java version "1.6.0_22"

    ReplyDelete
  12. A NoClassDefFoundError generally means that you're missing a library jar; in this case the package of the missing class is org.apache.lucene. Lucene is the text search engine library that OpenGrok uses to index your source code, and is the reason for its speed and search syntax.
    Unzip any jar file, and you'll find META-INF/MANIFEST.MF, which defines in the Class-Path entry the relative location of its libraries:
    Class-Path: lib/ant.jar lib/bcel-5.2.jar lib/jrcs.jar lib/servlet-api.
    jar lib/swing-layout-0.9.jar lib/lucene-core-3.0.2.jar lib/lucene-spe
    llchecker-3.0.2.jar lib/derbyclient.jar lib/derby.jar
    What this means is that you need a lib/ subdirectory adjacent to opengrok.jar with the jars listed, so make sure you include lib/ when you're copying opengrok.jar in step 3.

    ReplyDelete
  13. I would like to protect our code browser with user authentication mechanism. Could you let me know if I could achieve it with opengrok

    ReplyDelete
  14. I haven't tried that myself, but I don't believe OpenGrok has any authentication mechanism built-in.

    ReplyDelete
  15. Thanks Mike. It worked.
    I have another question. I am trying to index whole linux kernel which is in one of my mountable partition. For this I have created a symlink to the original linux-kernel directory. The indexing seems to be fine but when I browse I get "The requested resource is not available." Is symlink parsing support available in opengrok? When I copy files into my src directory everything seems to be working fine.

    ReplyDelete
  16. @Sujit - it looks like symlinks are supported with the use of the -N /path/to/symlink option, though I haven't tested it.

    ReplyDelete
  17. This is really useful Mike. One more question, how can I have multiple index of different source tarballs generated? Let say, I first generated the index for Linux kernel (and I have configuration.xml prepared). Now I want to download another source (for example freeBSD source) and generate the another index, how would i do that? If I use the above command which you mentioned, it will again generate the index for Linux source as well.

    ReplyDelete
  18. @godandme - the index generation process should ignore any projects that haven't changed since the last time it was run, so the index for your Linux kernel source will only be updated if your working copy has changed.

    ReplyDelete
  19. Hi Mike,
    Many thanks to your guide to install Opengrok. I successfully run a OpenGrok by following your instruction with some directory change to where i want.
    I installed it with OpenGrok 0.12.1, Oracle JDK 1.7 and tomcat7.

    ReplyDelete
  20. Hi Mike,
    I installed OpenGrok 0.12.1, Oracle JDK 1.7 and tomcat7.I changed some directory and web.xml config as well.When i deploy it's creating webapp in tomcat7.Now I try to search webapp from URL it showing HTTP Status 404 - /TEST_12.1/. Please help me to resolve the issue.

    ReplyDelete