History of the use of Python at STScI for science software

FITS Kernel

Two new instruments, Near Infra-Red Camera and Multi Object Spectrometer (NICMOS) and Space Telescope Imaging Spectrograph (STIS), were to be added to HST during Servicing Mission 2 (STS-82). The pipelines for the existing instruments that operated on a custom STScI format called GEIS (Generic Edited Information Set) that was similar to the FITS format, but which had the header and data in two separate files. Since STScI was required to distribute the HST data products as FITS files, and the time appeared ripe to make FITS a disk-based data format that the pipelines could process directly, rather than convert to FITS for storage in the archive and distribution to astronomers. STScI decided that the NICMOS and STIS pipelines would use the FITS format directly, as well be written in C using the above mentioned CVOS interface.

IRAF originally supported one data format, called the "imh" format, which also used two different files. When STScI became involved, support was added for reading and writing the GEIS format. The I/O modules for handling these formats were referred to as kernels. With the requirement to read and write FITS files, there became the need to add a FITS kernel. An early version had been developed at STScI that only supported simple forms of the FITS file, i.e., FITS files without extensions. The FITS standard had added the capability to have additional data elements in the file, called extensions, with each extension looking much like the original data element in that they consisted of Header/Data Units (better known as HDUs). Extensions could contain Image data, ASCII tables or Binary tables. This added complication to the kernel, as well as figuring out the appropriate user interface for specifying which extension was to be accessed, and where the kernel would look for an image if no specification of which extension was to be used (technically speaking, the first HDU in the file was called the Primary HDU, and its header was slightly different than those of the extensions).

Since this involved changes to the IRAF system, discussions were carried out with NOAO on two important points during the spring of 1995: 1) Could the FITS kernel be included in an IRAF release by the time of the servicing mission (approximately two years later)? We were assured that would be no problem. 2) Could we work out the user interface to the specification of extensions for the FITS file, and how metadata in the primary HDU would apply to extensions. At that time an agreement with NOAO was developed for the user interface, and work at STScI proceeded on that basis.

At subsequent IRAF Technical Working Group meetings, held twice a year at the AAS, we asked if the kernel would be released in time. For the June 95, January 96, and June 96 meetings we were assured that it would be. At the January 97 meeting (barely a month before the actual launch), we were told that it would not be released in time, and probably not until late summer or early fall. This was received by STScI management with a great deal of unhappiness as it meant STScI would be responsible for releasing an alternate version of IRAF until the FITS kernel was included. Not only was this extra work for STScI, it seriously complicated installations for the users if they also wanted the normal IRAF installation.

To add to the consternation, STScI was informed at the close of business the Friday before the June 97 TWG meeting the following Monday that the user interface was to be changed. There was no consultation about the change, as it was a fait accompli.

At this point there was great frustration at STScI in the late 1990's and a feeling that we were boxed into a very closed environment with a great deal of existing software that required it but with little influence on the core system. How to escape became the key issue.

Writing a completely new system appeared out of the question. It would take many years to replicate the existing software with existing staff, and besides, we would have to continue to write new software for the old system to support new HST instruments and requirements. Furthermore, efforts elsewhere to replace old astronomical software systems were not going well (our primary example being the problems of replacing the radio astronomy interferometry software system called AIPS with one completely rewritten in C++ called AIPS++). Some more gradual switch was needed, but that seemed very difficult to accomplish.

A Possible Escape Route

The IRAF CL was a particularly vexing problem in that it prevented more extensive use of scripting because of a number of shortcomings, such as the near impossibility of using script subroutines or functions due to a common namespace and a lack of useful error handling (at that time) or diagnostics. If we could develop a better CL, it would be a significant improvement in its own right. But even more important, if the new CL was run outside of the IRAF VOS, it could provide a means of integrating software written outside of the VOS into a common user interface allowing a gradual transition away from IRAF. In other words, we would retain access to all the old tools and write new tools in a new software environment and at the same time, users would be able to run both in an environment that was common to both. The idea of developing a better CL for IRAF was not a new one and had been floated for at least a few years prior to this, but with no serious study or action. This idea of using a new CL for the purposes of providing a way out started being considered in early 1998 and I started advocating it to the branch in the spring of that year (at a retreat held at the Turf Valley Resort). The branch lead, Rick White, was very supportive of the idea.

Language Choice

How to implement this idea percolated for several months. The biggest issue was what existing scripting language to adopt. We never considered a new custom scripting language seeing that developing a good one was difficult and required a lot of work, for which we had no resources to support. At that time, there appeared to be 4 possible choices: Perl, Tcl, Python, and IDL.

Alternate CL Progress

Improved array package

At the Long Beach Python Conference in January 2001, there was a BoF session regarding scientific Python tools. It was fairly widely agreed that Numeric needed significant improvement and probably needed to be rewritten. At that time Paul Dubois had been involved in that area in Python for years. He funded David Ascher to produce Numeric documentation as well. At that conference we met two significant future contributors in scientific python, Travis Oliphant and Eric Jones, who had already started the scipy effort. At the BoF it was mentioned that people had talked to Guido van Rossum about what to do about Numeric and he recommending writing the main framework in Python with the inner loops in C. Seeing as no one else was about to start such an effort we decided to do so. I, by hook and crook, managed to devote almost two solid weeks to prototyping an approach along the lines that Guido van Rossum suggested and determined it was practical. Its main drawback was that it increased the overhead of setting up array thus it would make small array operations slower.

I handed off the development of this to a new staff member, Todd Miller, who developed the full package, which we called numarray. It had many improvements over Numeric, which are detailed in this reference (https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.112.9899). We started developing other packages based on it, particularly pyfits. The first release was early 2002).

Numarray began attracting users, though disappointingly, the scipy project never found time to switch, and for a number of users, the overhead increase for small arrays was an obstacle. Even so a new and very useful image processing package was developed for it instead of Numeric. This led to concern on the part of Travis Oliphant that there was a split in the community. He wished to see a pure C package to eliminate the Python overheads. When we started, the only way to have an object oriented array package was to write it in Python since Python at that time did not support full class objects created in C. That had changed with Python 2 and at that point it was possible. Unfortunately we did not have the resources to rewrite it yet again. So Travis decided to rewrite it himself to address the needs of both communities. This is the effort that led to numpy.

Fortunately, early in this effort while he was still designing how it would work we were both invited to a meeting at Berkeley in March 2005, where a proposal for funding of a Python-based fMRI software project was being prepared; they were concerned about the potential split in the community. I assured them that we would use numpy once it met our needs. But far more important was the opportunity it gave Travis and me to discuss these issues. I explained how numarray handled all the sticky issues in detail, and it turned out Travis adopted many if not all of the same solutions for numpy. Much of this discussion was on a drive to and from San Mateo to have a meeting with Paul Dubois and Guido van Rossum over lunch. The issue that was raised there was the possibility of having numpy included in the Python Standard Library. Instead, Guido convinced us that would be a bad idea. And the result instead was settling on an array interface protocol that would allow other packages to support this protocol, wherein it would faciliate numpy interchanging data with these packages as arrays. This turned out to be quite useful, besides Guido being quite right about keeping it out of the Standard Library.

When numpy became mature, we began the process of migrating our software to use it instead of numarray. Strategically, it was a big win since we got all the functionality we needed, without having to provide support for it, as well as being a much better package for the community in replacing Numeric, despite the pain of switching.

PyFITS

Once we had sufficient functionality in numarray, we adopted Paul Barrett's pyfits package as the basis for enhancement. Numarray, and later numpy, allowed us to access binary tables without having to split columns into separate arrays. We could access the original memory (well, with caveats), by row, column or both without requiring copying the data buffer. This need drove the need for structured arrays in numarray and numpy and has become a heavily used feature in general. Several people have worked on pyfits, which is a complex package, largely due to the quirks of the FITS format that requires much code to deal with. These involved Paul Barrett, J. C. Hsu, James Taylor, Chris Hanley, and Erik Madison Bray at different times before it became part of Astropy. A significant amount of work was needed to switch pyfits from numarray to numpy when numpy became available. We were aided in this effort by Travis Oliphant.

Once at a sufficient level of maturity, we could begin to write applications code in Python using numarray and pyfits. The applications and libraries that were developed as a result are listed later.

Plotting and Image display tools

Without an adequate plotting package, Python would not be useful for interactive data analysis, and our ultimate goal was that astronomers would find it as useful as IDL for that purpose, i.e., instead of relying on IRAF applications that had graphics output, that they could generate arbitrary graphics output (including images) and ultimately generate publication quality output for paper submission. Unfortunately, as mentioned, no existing package satisfied all the requirements (quickly: open source, works on all OS platforms, didn't force use of a specific GUI framework, publication quality output, and image support). Satisfying these requirements was no simple matter. See the paper by Michele de La Pena on a survey of existing packages at the time (http://www.aspbooks.org/publications/281/193.pdf)

When at the 2001 ADASS meeting in Victoria, BC, Canada, we discussed with Eric Jones this particular problem. He had just co-founded a new company, Enthought, and had been hired to give a tutorial on scientific Python. Since we both had similar needs, STScI entered into a contract with Enthought to produce such a package (in actuality, it was only partial funding since they had other contracts that could use the same capabilities). This effort became the Chaco package (and Traits was a byproduct of that effort, at least out of my suggestions for how to handle plotting parameters and constraints on them).

At the time we were based mostly on Tkinter for the GUI framework (mainly because it was part of the Standard Library thus avoiding potentially serious installation hassles, which were a big deal then, much worse than now). Enthought was mostly focused on WxWindows (as called at the time, now WxWidgets). Over time we weren't seeing the kind of tools we had hoped to see, and nothing was working on Tkinter. When I asked for a simple plotting example, what I got instead was a 3000 line example, a far cry from a few line plot example I was hoping for. At that point I was beginning to realize that our needs were not that similar. Their needs involved a display of several plot windows where interactive changes in one would be reflected in all the others as they were often different views of different quantities sharing common axes. This was not a very serious need of ours. We mainly wanted simple plots without such interactive sophistication.

I was finally invited to go to Enthought (in Austin, TX) for a week to get a Tkinter version working (November 2-7, 2003 ). That did not succeed, perhaps due to my lack of skill, but more worrisome was that I could see much internal communication going on with this project that was completely invisible to us away from their offices. Furthermore, their scope had grown considerably as they seemed to be in the process of layering their own widgets on their plotting scheme. When returning back to STScI I was disheartened to find that even the Windows WxWindows version of their system would not install. Not long after that point I made the decision to abandon Chaco, figuring we would have to do our own plotting package (email shows this to be finally decided around January 19, 2004).

I announced our intent to do so on the mailing list right after that, which resulted in one email that asked why we weren't considering matplotlib. It sounded vaguely familiar, and I searched my email for it. In the original announcement it was limited to one GUI framework and clearly didn't satisfy our needs. On the other hand, a more recent release seemed to be broadening its coverage of our requirements (still short of our requirements), however, but it looked like it may be promising with more work. So I contacted John Hunter shortly afterwards to see if he was willing to meet the requirements, particularly if we helped in the effort. He was quite willing (only after his passing did I find out that my request he got approval to work full time on it for a while, https://www.youtube.com/watch?v=e3lTby5RI54). It was one of the best decisions I ever made.

At that point things started to move quickly on matplotlib. A major issue was easily supporting different GUI frameworks. I suggested that he adopt the solution that Chaco had adopted, namely a generic package for rendering graphics in memory and then blitting them to the GUI as an image. That had a performance penalty, but made it very general. At first John tried a different package than what Chaco was using, but within a few weeks, the switched to the same one, namely Anti-Grain Geometry (AGG,https://en.wikipedia.org/wiki/Anti-Grain_Geometry), and that proved very useful.

Other improvements needed were image display, contour plots, and math display. At this point more of our staff became involved including Nadia Dencheva for contour plot support, Todd Miller for the TkAgg backend and numerix module for handling a portable interface to both numeric and numarray, and Paul Barrett for font rendering. Some time later we hired Mike Droettboom and he was given the job of improving the math display and eventually becoming the major contributor to the internals including revamping how transforms to coordinates were handled. John Hunter eventually designated Mike as his replacement when he was no longer able to head the project due to his illness.

We encouraged John to give a talk at pycon and funded his travel, as well as having him give a talk on matplotlib at STScI. We coauthored a paper he presented at pycon called "Matplotlib: from brain surgery to rocket science" (https://slideplayer.com/slide/5175304/) with a somewhat shocking early slide that apparently caused some in the audience to have their brains lock up when seeing it.

Distribution and Installation

A long-standing issue in Python is the ability to distribute packages easily. Distutils was developed early (as a matter of fact, it originated out of discussions at that Houston conference).

The handling of distributions changed a number of times through the development. At one point we joined with Gemini to generate a Sage-like package called Ureka. This was supplanted by Conda-based tools when they became mature. And now things are moving more towards pip.

Regression Testing

Previous to PyRAF, it was very difficult to construct a regression test system for IRAF tasks. Using the CL itself to script such tests was fragile since there was no mechanism to handle failing tests (we could have started up the IRAF environment for every test, but that would impose a large overhead). However, with PyRAF, we now had a means of handling testing errors robustly, and thus we began slowly constructing regression test for all our IRAF-based software. This was to prove very useful, particularly with new IRAF releases (NOAO also did not have any regression test system and more than once we were able to catch problems with IRAF releases that they did not). The initial construction of this regression test system was started by Howard Bushouse.

Did I say three?

Gemini

In 2002 I was asked to organize an AURA-wide workshop on the future of IRAF/Data Analysis Software within AURA (Association of Universities for Research in Astronomy) by Bill Smith, the president of AURA. (Actually the request was made higher up, and the task was passed down to me). I found that NOAO had a lot of resistance to setting up such a workshop, mainly because they felt it was premature. I finally reported back that I didn't think they were ready for such a workshop, and that perhaps Bill Smith should ask the relevant AURA Centers for a white paper on their thoughts and plans for the future of data analysis software. Eventually, after a number of months of delay, the request worked its way down to me again, and I set up a committee that wrote a lengthy white paper, which was sent back to AURA (I must relate with some amusement that when giving the white paper to someone in the Directorate I was told it wasn't what Bill Smith wanted; my explanation that I was the one that told Bill what to ask for apparently was not believed.)

A few years later AURA arranged to hold a two day meeting on that topic just before the AURA board meeting tasked with selecting the new Director (April 12, 13 2005). The meeting had people from NOAO, Gemini, and STScI present (at least 3 from each institution), as well as a couple outside astronomers to present the outside perspective. The first day proceeded with each institution giving a status of their plans, and the outside astronomers giving their comments. It was clear that NOAO had different factions with competing viewpoints, but generally those at the meeting were driven primarily by the IRAF perspective (I later found out that NOAO never did provide a white paper response). That afternoon the IRAF group listed out work they felt was needed to improve IRAF. Among that work were three important items that I recall. The first was the need to support 64-bit code, the second was that the change in the memory model for the Gnu compiler broke IRAF, and IRAF continued to work only with a legacy mode switch, for which it was uncertain how long that would be supported. The last was a resurfacing of the "Multi-language CL" proposal, for which a brief digression is warranted.

That proposal first surfaced at the last IRAF Technical Working Group meeting in August 2001 at NOAO (at that point all other organizations had stopped coming thus there was no longer a need to meet at the AAS). The proposal was to take a different approach for supporting Python, as well as other scripting languages, in particular by subsuming them into the IRAF CL. That was unacceptable to us since there were numerous sticky technical issues had not been addressed that could cause it to fail, but even more important, it meant IRAF would control when versions of Python would be updated that was simply unacceptable.

At that point in the meeting, now in the late afternoon, it was asked if Gemini and STScI were on board with that plan. To which I responded that these were nice things for IRAF, but not particularly of interest to STScI since we were already on a quite different path and would not have resources to help in their effort. The result was as though I had ruined the party and there appeared to be several moments of stunned silence. And then something utterly astounding happened for which I did not think was possible.

To understand that requires another digression. In the morning the Gemini report suggested they were happy with how things were going with IRAF. As part of US participation in Gemini was the commitment by AURA for NOAO to provide calibration pipelines for Gemini. For whatever reason, that never happened and I knew that there was quite a bit a tension between Gemini and NOAO as a result. Nevertheless, it had not surfaced...until this point Suddenly, the Gemini team announced that they were very interested in adopting PyRAF for their system, and then that atmosphere of the meeting transformed immediately. The moderator (Jay Frogel) from AURA seemed very pleased at this development, and the next day was set for some informal discussions, primarily between whether the Multi-Language CL and PyRAF was most suitable for future development. The working group formed the next morning showed that all the same problems noted at the first proposal of the Multi-Language CL were still as significant, and it was clear to Gemini that it was not a practical approach (besides, PyRAF existed, and the other did not, not even as a prototype).

Unfortunately this left much bitterness in the NOAO IRAF group, and even with subsequent attempts to involve them in the Python effort, they would have none of it.

The result of the Gemini interest is that we became involved in helping them address their pipeline problems. With the lack of pipeline software from NOAO, their pipelines were developed in CL. Since the CL does not support subroutines in any practical way. Each pipeline typically was thousands of lines of code in one file. Since the CL had no error handling or even useful error messages, all inputs had to be carefully checked leading to much code of that nature. When new instruments were added, code from other instruments pipelines would be cut and pasted into the new one. The result was a repository of tens of thousands of lines of CL code, with a lot genetic similarity, but not identical. It had become a maintenance nightmare.

They wanted to have all this run under PyRAF so that they could begin to change the structure of the code. But to start it needed regression tests to ensure that the PyRAF results matched the IRAF ones. They worked out a contract with STScI for us to provide a suite of regression tests (most of which was developed by Robert Jedrzejewski). This also served as an acid test for PyRAF and a number of improvements to PyRAF resulted. It also led to a productive and long-term working relationship with the Gemini developers, as well as a broader AURA acceptance.

Image Display to DS9

Although matplotlib provided the capability of displaying images, it wasn't a substitute for the capabilities of DS9. Being able to display images to DS9 from Python was an important goal. This was possible using the NOAO Client Display Library developed in C by NOAO as part of the OpenIRAF project (pretty much the only usable product from that effort: http://iraf.noao.edu/irafnews/apr98/irafnews.d.html). The irony is that it enabled an aspect of our escape from IRAF.

Multidrizzle

Multidrizzle was probably the first very extensive application developed in Python at STScI. The initial version was written by Anton Koekemoer and then extended by the Science Software Branch, initially by Chris Hanley. It essentially managed the use of the then Fortran drizzle program (since converted to C) for multiple overlapping images to resample them onto the same grid. Its user interface was quite complex with many user options depending the details of the set of images to combine. This complexity drove the need for the next tool.

GUI parameter editor

The number of possible parameters for Multidrizzle became more than could be easily managed by a simple epar task, and as a result a more capable Tkinter-based tool (eventually called Teal) was developed (primarily by Warren Hack) to allow an easier to use interface that lessened the need for users to consult the documentation through the use of drop down menu choices, radio buttons and checkboxes. The number of possible parameters is over 70!

CALCOS

When our work on Python infrastructure was ready to develop calibration pipelines, there were only two more new instruments planned for installation in HST, Wide Field Camera 3 (WFC3) and the Cosmic Origins Spectrograph (COS). Since WFC3 was using detectors very similar to Advanced Camera for Surveys (ACS) and Near Infra-Red Multi-Object Spectrograph (NICMOS), the decision was to make its calibration pipeline based on the code of those two instruments, leaving only one new pipeline to develop. The CALCOS calibration pipeline was the first written in Python, and was successfully developed and used for that instrument when it was launched.

HST Exposure Time Calculator

Exposure Time Calculators have had a interesting history. Early in the HST effort, there was no official support for such simulation tools, but very many staff wanted them. A few of them spontaneously were developed here and there, some as generic synthetic photometry tools, some that were instrument specific. At some point the STIS team at STScI developed a web based tool written in C. But before long they found it was a maintenance hassle and it was given to our group to maintain. It ended up being rewritten.

Not long after that there was a Goddard/STScI effort to produce a GUI-based tool that integrated the proposal planning tools with the ETCs. A demo prototype was developed for the Advanced Camera for Surveys (ACS) that was written in Java. It eventually got the attention of the Director, and he directed that such a tool be made for HST. After some months and being unhappy with the lack of a decision on how to proceed, he also directed that they should base it on the prototype (called the Science Expert Assistant), though it was renamed to the Visual Target Tuner and ETC. It was considerably more complex than the CGI-based ETC that existed. After well over a year of development, some staff astronomers realized that the ETC component was not web based, and they directed that it should support a web interface as well (the rationale, not unreasonable, is that while people that have had a proposal accepted would not find it objectionable to install an application to support their observations, those just proposing may find it objectionable). Nevertheless it resulted in major changes to the design and implementation. The GUI application version eventually atrophied, but leaving much of the GUI-based structure behind. All this work was done in a different branch from ours.

This is all relevant since after a few years of use, the project was transferred to our branch to maintain. Suddenly all hell broke loose at the next Proposal Deadline (where web usage spikes greatly). The server would crash every few minutes for the last two days. This led to making a priority the ETC server more robust for the next deadline a year later. Nevertheless, it was just as unstable, and the decision was made to start a new effort with a Python-based ETC. This was a sizable effort to get it working with a hard deadline in place. The new ETC had to agree wtih the results of the previous one, where there were tens of thousands of regression tests to pass. The effort was led by Vicki Laidler and the team working with her (Ivo Busko and Megan Sosey) met the deadline wtih a reliable ETC. The effort reinforced my view that any complex software isn't really well tested until the results are compared with a completely different implemention.

The Management Environment during this Era

The timing of the development of PyRAF and the subsequent library work occurred at a favorable time with regards to upper management oversight. This is due to a couple of factors:

• HST was in routine operation. Much of the overhead of NASA reporting requirements for software development had mostly gone away. Well prior to HST launch, there were many procedural and documentary requirements for the development of calibration software. Very early in the project, much "PDL" (Program Design Language) design documents were written outlining the software to be developed. A great deal of that effort appeared to be pointless in the long run. Shelves of such design documents were written and eventually put into storage. I'm not aware of any attempt to use them after they were written.

  As launch loomed closer (but not as close as it actually was), panic struck many areas of software development, and the focus became more on actually writing and testing software, than writing about writing software. As the calibration pipelines started being used, the management overhead was relaxed to a significant degree. For new instruments, there were still requirements to produce PDRs (Preliminary Design Reviews) and the like, but not much for data analysis tools, which were much more informally managed.

• Severe unhappiness with the IRAF situation. In particular the problems with the FITS kernel delays and other difficulties led much of the upper management happy to see any way out of the lack of control over the software environment for the calibration pipelines and data analysis tools. Any sign of progress on this front was greatly welcomed, and there did not appear to be any great desire to manage it in detail while good progress was being made.

As a result, most of the work done on PyRAF, pyfits, numarray, matplotlib and the other tools had minimal management oversight. Other than regular status reports, there was no requirement to submit detailed plans, requirements, schedules on short time scales. Planning documents were high level, and usually done once or twice a year. During this more than a decade of work on Python tools, we had a very free hand with minimal interference.