Release Management

This page describes how we release software.

The following pages might be of use:

  • Development Tree
  • Testing
  • Building a release

    Splitting the tree

    1. Look for auto directories that don't have corresponding junit/ directories.
      cd $PTII
      ls -d `find . -name auto | awk -F / '{for(i=1;i<(NF);i++) { printf("%s/", $i);}printf("junit\n")}'`o
      
    2. Check that each demo has a test:
      cd $PTII/ptolemy/configs/doc
      make check_tests
      
    3. Add any new models to $PTII/ptolemy/configs/doc/completeDemos.htm
    4. If necessary create applets for new demos
      Run
      cd $PTII/ptolemy/configs/doc
      make check_applets
      
      Note that not all the applets listed are missing. Some of them use the serial port, JAI, JMF etc.

      To add an applet:

      cd $PTII
      make install
      cd $PTII/ptolemy/domains/wireless/demo/SmartParking
      $PTII/bin/copernicus -codeGenerator applet -targetPath ptolemy/domains/wireless/demo/SmartParking SmartParking.xml
      
      Then add the .htm files to the makefile.
    5. Ptolemy handled about:copyright URLS specially. One facility on the copyright page is the ability to expand all the model, .html files and .pdf files on the various demonstration pages. This allows us to verify that all the resources are present in the tree.
      Following the links on about:copyright is an essential part of testing each release.
      For details, see actor.gui.GenerateCopyrights and actor.gui.HTMLViewer
    6. Ptolemy II is organized in to jar files, where each directory contains a jar file of the .class files and another resources that are necessary for that package. These resources include files like images and pdfs. Parent directories contain the jar files in children directories.
      When building a release, it is usually best to build the Web Start release first because it is easier to change one jar file and retry something in Web Start than it is to rebuild an entire Windows installer.
      Use about:copyright to test the Web Start release.
    7. Check out a clean tree and build:
      # Check out a clean tree
      svn co svn+ssh://source.eecs.berkeley.edu/chess/ptII/trunk ptII
      cd ptII
      export PTII=`pwd`
      ./configure
      ant
      
    8. Clean the tree by running make checkjunk and adding files as appropriate to the makefiles:
      cd $PTII
      make clean_shipping
      make checkjunk
      
      Look for *** lines and add files to makefiles or add them to the clean_shipping rule in $PTII/makefile and the most recent src.ex file in $PTII/adm/gen-x.y/src.ex.
    9. Run $PTII/util/testsuite/chkjava on the java files and fix problems
      Shortcut: Run it only the most recent files added
      Create a list of java files by either running
      
      cd $PTII
      make cleanDerivedJavaFiles
      make clean_shipping
      
      Or removing the files by hand:
      cd $PTII/ptolemy/actor/ptalon
      rm `make echo_OPTIONAL_JSRCS`
       cd $PTII/ptolemy/data/expr
      rm `make echo_OPTIONAL_JSRCS`
      cd $PTII/ptolemy/data/unit
      rm `make echo_OPTIONAL_JSRCS`
      cd $PTII/ptolemy/copernicus/kernel/fragment
      rm `make echo_OPTIONAL_JSRCS`
      
      # Create a file that contains the names of the java files to include
      cd $PTII
      rm -rf vendors ptolemy/apps
      rm -rf ptolemy/domains/ct/demo/Corba/util/*.java
      
      
    10. Generate a list of files
      adm/bin/ptIItxtfiles > /tmp/f
      egrep '*.java$' /tmp/f | grep -v /vendors/ >& /tmp/j
      
    11. Check for missing makefiles:
      cd $PTII/adm/test
      make missingMakefiles
      
      Below are the missing makefiles that are ok to be missing:
      ./config/makefile
      ./doc/coding/templates/makefile
      ./ptolemy/backtrack/automatic/ptolemy/actor/lib/makefile
      ./ptolemy/backtrack/automatic/ptolemy/domains/sdf/lib/makefile
      ./ptolemy/backtrack/automatic/ptolemy/math/makefile
      ./ptolemy/backtrack/ui/makefile
      ./ptolemy/backtrack/util/java/util/makefile
      ./ptolemy/plot/servlet/makefile
      
    12. Check for directories that have java files but are not listed in doc/makefile:
      cd $PTII/adm/test
      make missingDocPackages
      
      Below are the packages that need not be in doc/makefile.
      config
      contrib.actor.lib.example
      diva.util.java2d.svg
      doc.coding.templates
      doc.tutorial
      doc.tutorial.domains
      doc.tutorial.graph
      doc.tutorial.graph.junit
      doc.tutorial.gui
      ptolemy.actor.corba
      ptolemy.actor.corba.CoordinatorUtil
      ptolemy.actor.corba.CorbaIOUtil
      ptolemy.actor.corba.util
      ptolemy.actor.lib.javasound.test.pitchshift
      ptolemy.actor.lib.jxta
      ptolemy.actor.lib.reactable
      ptolemy.actor.lib.tutorial
      ptolemy.backtrack.automatic.ptolemy.math
      ptolemy.backtrack.eclipse.ast
      ptolemy.backtrack.eclipse.ast.transform
      ptolemy.backtrack.eclipse.plugin
      ptolemy.backtrack.eclipse.plugin.actions
      ptolemy.backtrack.eclipse.plugin.actions.codestyle
      ptolemy.backtrack.eclipse.plugin.compatibility
      ptolemy.backtrack.eclipse.plugin.console
      ptolemy.backtrack.eclipse.plugin.dialogs
      ptolemy.backtrack.eclipse.plugin.editor
      ptolemy.backtrack.eclipse.plugin.preferences
      ptolemy.backtrack.eclipse.plugin.util
      ptolemy.backtrack.eclipse.plugin.widgets
      ptolemy.backtrack.test.array1
      ptolemy.backtrack.test.random1
      ptolemy.backtrack.test.test1
      ptolemy.backtrack.test.test2
      ptolemy.caltrop.ddi.util
      ptolemy.chic
      ptolemy.component
      ptolemy.component.data
      ptolemy.component.data.type
      ptolemy.component.domains.ptinyos
      ptolemy.copernicus.interpreted
      ptolemy.domains.gr.lib.quicktime
      ptolemy.domains.ptides.kernel.test.junit
      ptolemy.kernel.test.junit
      ptolemy.moml.jxta
      ptolemy.plot.servlet
      ptserver.test.helper
      ptserver.test.junit
      
    13. Add trailing newlines to files with $PTII/adm/bin/addtrailingnl
      cd $PTII
      # Run addtrailingnl without actually doing anything:
      cat /tmp/j | xargs $PTII/adm/bin/addtrailingnl -n
      # Run addtrailingnl with adding the new lines:
      cat /tmp/j | xargs $PTII/adm/bin/addtrailingnl
      # Run with -n and then without:
      cat /tmp/f | egrep '*\.[cjh]$' | xargs $PTII/adm/bin/addtrailingnl -n
      cat /tmp/f | egrep makefile | xargs $PTII/adm/bin/addtrailingnl -n
      cat /tmp/f | egrep README | xargs $PTII/adm/bin/addtrailingnl -n
      cat /tmp/f | egrep '*\.htm' | xargs $PTII/adm/bin/addtrailingnl -n
      cat /tmp/f | egrep '*\.tcl' | xargs $PTII/adm/bin/addtrailingnl -n
      cat /tmp/f | egrep '*\.xml' | xargs $PTII/adm/bin/addtrailingnl -n
      
    14. Convert tabs with $PTII/adm/bin/rmtabs
      cd $PTII
      # Run rmtabs without actually doing anything:
      cat /tmp/j | xargs $PTII/util/testsuite/rmtabs -n 
      # Run rmtabs with adding the new lines:
      cat /tmp/j | xargs $PTII/util/testsuite/rmtabs 
      # Run with -n and then without:
      cat /tmp/f | egrep '*\.[cjh]$' | xargs $PTII/util/testsuite/rmtabs -n
      
    15. Remove trailing spaces with: $PTII/util/testsuite/rmtrailingspace
      cd $PTII
      # Run rmtrailingspace without actually doing anything:
      cat /tmp/j | xargs $PTII/util/testsuite/rmtrailingspace -n 
      # Run rmtrailingspace with adding the new lines:
      cat /tmp/j | xargs $PTII/util/testsuite/rmtrailingspace 
      # Run with -n and then without:
      cat /tmp/f | egrep '*\.[cjh]$' | xargs $PTII/util/testsuite/rmtrailingspace -n
      
    16. Fix common problems in java files:
      cat /tmp/j | xargs $PTII/adm/bin/fix-files -n
      
    17. On a Unix box, check for files with control-Ms:
      cat /tmp/f | xargs $PTII/util/testsuite/controlm > /tmp/cms
      

      To fix the files, run:

      grep "has controlms" /tmp/cms | awk '{print $1}' > /tmp/cmfs
      svn propset svn:keywords "Author Date Id Revision" `cat /tmp/cmfs`
      svn propset svn:eol-style native `cat /tmp/cmfs`
      svn commit -m "Added keywords and eol-style properties" `cat /tmp/cmfs`
      
      You may need to use dos2unix to fix the files.

      Check keywords in all files:

      cd $PTII
      cat /tmp/f | xargs $PTII/adm/bin/svnpropcheck > /tmp/props
      grep keywords: /tmp/props | grep / | awk '{print $1}' > /tmp/keys
      grep eolstyle: /tmp/props | grep / | awk '{print $1}' > /tmp/eols
      cat /tmp/keys | xargs svn propset svn:keywords "Author Date Id Revision"
      cat /tmp/eols | xargs svn propset svn:eol-style native 
      svn commit -m "Added keywords and eol-style properties" `cat /tmp/eols /tmp/keys`
      
    18. Run $PTII/adm/copyright/fixcopyrights (not shipped with the release) and update the copyrights on the .java, .tcl and makefiles
    19. Run $PTII/adm/copyright/chkcopyright on all the nightly buildfiles.
      First, edit $PTII/adm/copyright/chkcopyright and update the year.
      Then, do:
      cat /tmp/f | grep -v .class | grep -v codeDoc | grep -v xml | grep -v \
      alljtests.tcl | grep -v .htm | grep -v qcf | grep -v .dtd | grep -v \
      gif | grep -v .svn | xargs sh $PTII/adm/copyright/chkcopyright >&  /tmp/copycheck 
      grep "does not contain a Copyright" /tmp/copycheck | awk '$1 ~ /.java$/ {print $1}' | grep -v /actor/corba >/tmp/jc
      
      Then, edit $PTII/adm/copyright/copyright.txt and $PTII/adm/copyright/insertcopyright and run
      adm/copyright/insertcopyright -p `cat /tmp/jc`
      
      and then
      adm/copyright/insertcopyright `cat /tmp/jc`
      
    20. Check out all the files again so we can build a release:
      cd $PTII
      svn update
      
    21. Run the nightly build, build a release:
      For each release, we copy the most recent $PTII/adm/gen-X.Y, update the version number and run make. See the top of the file for the appropriate make command.
    22. Check the copyright on new files First, untar the previous source release in ptII/adm/dists. Then:
      cd $PTII/adm/dists
      diff -r ptII8.0.1 ptII9.0.devel > /tmp/pt.diffs
      cd $PTII
      grep Copyright `egrep "^Only in ptII9.0.devel" /tmp/pt.diffs | \
         grep .java | sed 's@ptII9.0.devel/@@' | awk '{print substr($3,1,length($3)-1) "/" $4}'`
      
      grep Copyright `egrep "^Only in ptII9.0.devel" /tmp/pt.diffs | \
         grep .java | sed 's@ptII9.0.devel/@@' | awk '{print substr($3,1,length($3)-1) "/" $4}'` | grep -v `date +%Y` > /tmp/c2
      
      See $PTII/adm/copyright/fixcopyright, but basically:
      cat /tmp/c2 | grep Regents | awk -F : '{print $1}'
      
    23. Run $PTII/adm/bin/fixsince (not shipped with the release) and update the @since tags. To use fixsince, first edit $PTII/adm/bin/fixsince and update the variables at the top. Then
      cd $PTII
      $PTII/adm/bin/fixsince
      
      Then check the output. Then edit the file and uncomment the cp line at the bottom:
        #cp $tmpfile $file
      
    24. Run cd $PTII/doc; make chkpackagehtml (not shipped with the release) and create any missing package.html or README.txt files.
      cd $PTII/doc
      make chkpackagehtml > /tmp/s
      cd $PTII
      $PTII/adm/bin/mkpackagehtml `cat /tmp/s`
      svn add `egrep '(README.txt|package.html)' /tmp/s | awk '{print $2}'`
      svn commit -m "Added package.html and README.txt" `egrep '(README.txt|package.html)' /tmp/s | awk '{print $2}'`
      
    25. Concatenate all the javadoc files in to one file and run $PTII/util/testsuite/ptspell.
      cd $PTII/doc
      make docs
      cd codeDoc; find . -name "*.html" > /tmp/h
      cat /tmp/h | xargs cat > /tmp/docs
      cat /tmp/docs | aspell list -H --run-together-limit=10 | sort -u | 	sed -e 's/^\([A-Z]*\)\([A-Z]\)/\1 \2/g' | 	sed -e 's/\([a-z]\)\([A-Z]\)/\1 \2/g' | aspell list -H | sort -u
      
    26. Run findbugs and look at the results:
      cd $PTII/doc
      make findbugs.htm
      
    27. Consider using PMD or
    28. Use Eclipse to fix the imports of the entire tree.
    29. Use Eclipse to add @Override to appropriate methods.
    30. Consider fixing other Eclipse warnings.
    31. Use Eclipse Cleanup to clean the code
      1. Set up cleanup by going to the preferences window in Eclipse:
        (Windows: Window | Preferences)
        (Mac OS X: Eclipse | Preferences)
      2. Java | Code Style | Clean Up
      3. New, then type in a Profile name of "Ptolemy II" then hit OK
      4. In the "Profile 'Ptolemy II'" window:
        • Code Organizing: select "Format source code",
          select "Remove trailing whitespace",
          select "Correct indentation",
          select "Organize Imports".
        • Code Style: select "Use blocks in if/while/for/do statements"
        • Missing code: unselect "Add missing Annotations"
        • Unnecessary Code: select "Remove unused private members"
          select "Remove unused local variables"
      5. Save the cleanup settings by closing the Cleanup preferences window
    32. Consider using GCJ: The GNU Compiler for Java to report unused variables and other problems. Note that GCJ3.4 will fail to compile awt and swing classes, but it is still worth getting the other error messages.
      cd $PTII make clean make -k JAVAC=gcj JFLAGS="-O2 -Wunused -Wextraneous-semicolon -Wno-deprecated -C" all >& gcj.out & grep error: gcj.out | awk -F: '{for(i=3;i<=NF;i++) {printf("%s ",$i)} printf("\n");}' | sort | uniq | grep -v "not found in the decl"
    33. Consider building with the IBM JDK
    34. See Installers for how to build Windows installers. Building the Windows installer is fairly complex, below are some things to be aware of
    35. The Windows installer includes source files. These sources come from $PTII/adm/gen-N.M/ptIIN.M.src.jar. Obviously, this jar file should be created on a Windows machine so the end of line characters are Windows end of line characters.
    36. Look for files with bad permissions
      cat /tmp/f | egrep "(.java$|.htm$|.html$|makefile$|.tcl$|.xml$)" | xargs ls -l | awk '{ if ($1 ~ /x/) print $NF}' | grep -v chkjava > /tmp/execs
      cat /tmp/execs | xargs svn proplist
      cat /tmp/execs | xargs svn propdel svn:executable
      svn commit -m "Made regular .java files and other files not executable" `cat /tmp/execs`
      
    37. Tag the tree as the release tree and create a branch

      To make a branch for 8.0.beta, use the following command:

      svn copy svn+ssh://source.eecs.berkeley.edu/chess/ptII/trunk svn+ssh://source.eecs.berkeley.edu/chess/ptII/branches/rel-8-0-beta-2 -m "Creating Ptolemy II 8.0.beta branch"
      
      Then check out the release branch with
      svn co svn+ssh://source.eecs.berkeley.edu/chess/ptII/branches/rel-8-0-beta-2 ptII8.0.beta
      
      
      Note that you should avoid dots in the tag name
    38. Update the version numbers, see $PTII/makefile and $PTII/ptolemy/plot/makefile.
    39. Remove files that are not to be shipped from the release:
      cd ptII8.0.beta
      export PTII `pwd`
      ./configure
      make cvs_delete_clean_shipping
      
    40. The nightly build uses a makefile in ~ptII/adm (not shipped) to build the release. Copy the most recent makefile and set up a new tree for the release tree.
    41. We build a separate src.tar.gz file under Unix so that the line endings are correct.
      svn co svn+ssh://source.eecs.berkeley.edu/chess/ptII/branches/rel-8-0-beta-2 ptII8.0.beta
      cd ptII8.0.beta
      setenv PTII `pwd`
      ./configure
      make fast install
      cd $PTII/adm/gen-8.0
      make USER=cxh PTIIHOME=/home/cxh/src/ptII8.0.beta PTIIADM=/home/cxh/src/ptII8.0.beta/adm TAR=/bin/tar JAR=/usr/java/default/bin/jar clean all src.jar release_sisyphus
      
      
      The resulting file with the .class files gets copied to the website and untar'd. Note that this does not require Java 3D on the build machine. Java 3D is needed for the applets, we copy the jars over later.
      make USER=cxh PTIIHOME=/tmp/cxh/ptII8.0.beta PTIIADM=/tmp/cxh/ptII8.0.beta/adm TAR=/bin/tar JAR=/tmp/cxh/jdk1.5.0_09/bin/jar update_ftp
      
      or
      make USER=cxh PTIIHOME=/tmp/cxh/ptII8.0.beta PTIIADM=/tmp/ptII8.0.beta/adm TAR=/usr/sfw/bin/gtar JAR=/bin/jar update_ftp
      
      1. Fix the links to ptplot downloads:
        cd /export/home/pt0/ptweb/ptolemyII/ptII8.0/ptII8.0.beta/ptolemy/plot
        ln -s ../../../../../java/ptplot5.7/ptolemy/plot/ptplot5.7* .
        cd doc
        rm -rf download
        ln -s ../../../../../../java/ptplot5.7.beta/ptolemy/plot/doc/download .
        
      2. Copy gr.jar from a Windows box to ptolemy/domains/gr
      3. Copy codeDoc.jar from a Windows box and unjar it
    42. We build the webstart jar files under Windows
      cvs -d :ext:source:/home/cvs co -r rel-7-0-beta-2 ptII
      cvs update -P -d
      mv ptII ptII8.0.beta
      cd ptII8.0.beta
      export PTII=c:/tmp/cxh/ptII8.0.beta
      
      Set up missing packages such as joystick, JAI, JMF, javacomm, quicktime
      Use an older javac so that the .class files are compatible.  For
      example,  use java 1.4.x so that java 1.4.x and 1.5.x can use the
      .class files. 
      
      export PATH=/cygdrive/c/j2sdk1.4.2_12/bin:${PATH}
      java -version
      ./configure
      make clean fast install >& make.out
      
      Copy ~ptII/adm/certs/ptkeystore to $PTII.  Note you will
      need access to the ptII account on bennett 
      
      make jnlp_dist STOREPASSWORD="-storepass xxx" KEYPASSWORD="-keypass xxx"
      
      xxx is the ptkeystore password.
    43. Build the webstart files. This is complex . . .
    44. Create the webstart and installer files.
      cd $PTII
      make jar_dist
      
      Look for missing jar files in the output.
    45. Create the src.jar file:
      cd $PTII/adm/gen-8.0
      make USER=cxh PTIIHOME=/cygdrive/c/cxh/ptII8.0.beta COMPRESS=gzip TAR=tar clean all src.jar 
      
      Below is the command to compare the old tree and the new tree:
      diff -r -I '$Id:' -x alljtests.tcl -x allsimplejtests.tcl -x '*.jar' -x codeDoc -x '*.out' -x .svn -x '*.class' ~/ptII ~/src/ptII8.0.beta >& /tmp/pt.diffs
      
    46. Under Windows, build the installer.
      cd $PTII
      make exes
      cd $PTII/adm/gen-x.y
      make setup
      
    47. Test the installer

    Last Updated: $Date: 2012-10-02 02:00:26 +0200 (Tue, 02 Oct 2012) $