
Notes on the MontagePy Package


Layout
------
The top level directory for the package is "MontagePy".  See the diagram 
below.

In it is another subdirectory of the same name ("MontagePy") which will
contain the files that make up the package distributed to the Python Package
Index (PyPI).  This doubly nested structure is normal for Python packages.
The outer top level directory name is immaterial to the build but it would
normally be silly to use something different from the package name.

The "MontagePy" subdirectory initially contains "__init__.py", a
file that tell Python that this is a package.  It can also contain the
package description (available using help()) and setup code, though we
don't need the latter. The other files necessary to make that package will 
copied in from elsewhere or built by Cython as described below.

This top level directory mainly contains the build instructions ("setup.py").
This is a simple Python script that provides metadata about the package.

|
|  Note: The descriptive metadata can include the following parameters:
|
|     name              name of the package
|     version           version of this release
|     author            package author’s name
|     author_email      email address of the package author
|     maintainer        package maintainer’s name  short string
|     maintainer_email  email address of the package maintainer
|     url               home page for the package
|     description       short, summary description of the package
|     long_description  longer description of the package
|     download_url      location where the package may be downloaded
|     classifiers       a list of classifiers
|     platforms         a list of platforms
|     license           license for the package
|


For MontagePy, this metadata also includes the C library and Cython wrapper
dependencies.  It has a couple of active bits:  it checks for the existence
of the Montage/lib and Montage/MontageLib directories; and it generates the
"extension module" parameter by running cythonize() on the .pyx files (below) and
list of libraries to create the linkage between python and the Montage C 
libraries.

The files needed to build the Cython wrapper are "parse.py" and in the "templates" 
directory.  Basically, parse.py looks through the Montage source directories for
a set of JSON files we have created that describe each module and creates a 
Python 'functions' structure with the information.  Then it uses the templates
to create the following files in the "MontagePy" subdirectory:


   templates/template_main.pyx  ->  MontagePy/main.pyx
   templates/template.pyx       ->  MontagePy/_wrappers.pyx
   templates/template.pxd       ->  MontagePy/wrappers.pxd

 
    main.pyx  --  .pyx files are compiled by Cython to .c files, containing 
    the code of a Python extension module. The .c file is compiled by a C compiler 
    to a .so file which can be import -ed directly into a Python session.  


    _wrappers.pyx  --  Contains the code describing the calling syntax for each 
    Montage module.  This conde is converted by Cython into C that can be called
    by Python.  Here is the snippet for the Montage "mAdd" function.


    wrappers.pxd  --  In addition to .pyx source files, Cython uses .pxd files 
    which work like C header files – they contain Cython declarations which are
    only meant for inclusion by Cython modules. A pxd file is imported into a 
    pyx module by using the cimport keyword.  In our case, the declarations 
    involve the names and calling parameters for each Montage C library module.


The creation of the C code associated with the above (main.c and _wrappers.c)
and the compiled shared-object files from these (the .so files are what actually
gets distributed) is done by cythonize() commands in setup.py.


The package layout looks like this:

   MontagePy
      |                                        
      | setup.py                              (Package setup script. 
      |                                        Besides defining the package
      |                                        this also runs the cythonize()
      |                                        function that populates the
      |                                        MontagePy directory.)
      |                                        
      | parse.py                              (Run manually before distributing                                       
      |                                        to create the pyx/pxd files from
      |                                        the templates.  Scans MontageLib
      |                                        for JSON module definitions.)
      |                                        
      | validate_json.py                      (A tool used during development                                         
      |                                        to find and check the validity  
      |                                        of the module JSON files in     
      |                                        MontageLib directories.)
      |                
      | lib                                   (This directory contains all the 
      |                                        support library .o files and was 
      |                                        populated by the MontageLib      
      |                                        Makefile.)
      |                
      |                                        
      | MontagePy |
      |           | __init__.py               (Package initialization file. In
      |           |                            our case this only contains the
      |           |                            package description text.)     
      |           |
      |           | wrappers.pxd              (The first three files here
      |           | main.pyx                   are generated before distribution
      |           | _wrappers.pyx              using a "parse.py" utility.)
      |           |
      |           | main.c                    (The C versions are generated
      |           | _wrappers.c                from the above using cythonize()
      |           |                            during setup.)
      |           |    
      |           | main.so                   (As are the compiled library 
      |           | _wrappers.so               versions.  Only these files are
      |           |                            distributed, the above five are
      |           |                            not.) 
      |           |    
      |           | FreeSans.tff              (The mViewer module required a
      |                                        font file and we include one rather
      |                                        than trying to find it on the user 
      |                                        machine.)
      | 
      | templates |
      |           | template.pxd              (These templates are used
      |           | template_main.pyx          by the "parse.py" script
      |           | template.pyx               to generate the pyx/pxd
      |                                        files above.)
      |
      | dist                                  (This directory gets created and
                                               filled when setup.py is run.  The
                                               resulting .whl files get uplaoded
                                               to PyPI for public consumption.)  



Build
-----
Building the MontagePy package consists of running "python parse.py" to build 
the Cython wrapper code and "python setup.py build_wheel --inplace", or variants
on this command.   Our setup code requires Python packages "Cython" and "jinja2"
which need to be pip-installed beforehand.  The top-level directory incudes a
"make.sh" file with the commands to produce a clean package and upload it.  This
script removes previous attempts from the package index and uninstalls the package
locally in preparation for the new attempt.  It also has code to tweak one of 
the functions in the cythonized _wrappers.pyx file:  We needed to include a 
reference in the Python part of the code there to our included font file.

But first we describe below how we built the package and our test "package index" 
setup and finally the process of getting it into PyPI.


                            . . . . . . 


Ultimately, we want to upload this package to PyPI for others to use but since
our initial trial and error phase will involve a lot of false starts we don't
really want that to be too public.  So instead we create a package index service
of our own, which is reasonably simple as there are a number of simple server
packages out there.  Here's the one we set up:

|
|   On desktop Mac (local machine "mosaic"), in directory: /Users/jcg/PyPIserver
|   
|      pip install pypiserver
|      rehash
|      mkdir packages
|      pypi-server -p 9022 -P htpasswd.txt packages &
|
|   This can be accessed through pip but also viewed through a 
|   browser at 
|   
|      http://mosaic:9022
|   
|   Once we have a package (on exodev in Montage directory) we
|   can upload it to this server using setup.py. (On ours, the 
|   password is "p....s".)
|


                            . . . . . . 


The processes of building the package, uploading it to a package index,
and installing it locally (though we don't do the last this way) are all
handled by "setup.py".  This file is a python script but one that mainly
calls the setuptools.setup() method.  This method is given all the package 
metadata (e.g. "author", "version") and does all the work of compiling 
building distribution files, installing and uploading to a package index 
for others to get.

You activate these steps with commands like


   python setup.py build
   python setup.py bdist_wheel upload -r http://mosaic.ipac.caltech.edu:9022


In the last command we uploaded it to our local package index server using
the "-r" flag.

To actually install the package from our index we use the standard 'pip'
command (with the alternate index again called out):


   pip install -i http://mosaic.ipac.caltech.edu:9022 MontagePy
 

However, since we ran the package index server without secure HTTPS we
need to authorize the client pip to use it.  This is done by adding pip
configuration (the ~/.pip/pip.conf file):


   [global]
   extra-index-url = http://mosaic.ipac.caltech.edu:9022
   [install]
   trusted-host = mosaic.ipac.caltech.edu



Use
---
The Montage package is called MontagePy but due to the Cython machinations
we need to refer to modules used as coming from "MontagePy.main":


    from MontagePy.main import *


After that, we can use it just like any normal Python module, e.g.:


    help("MontagePy")

    help(mHdr)

    mHdr("M51", 0.5, 0.5, "region.hdr")



