pyAstroStack: Getting prepared for first release

...where getting prepared means a LOT of work. By number of code lines maybe even more than there already is.

Things I'm working on, one by one:

Project file

I want the stacking process to be possible to continue on a later time starting from any point. I also want to make possible for example do the stacking couple times on different settings (perhaps to test which method works the best).

I'm implementing this via project file. It holds all the necessary information about source photos and what's done with them. Program reads the project file and it can tell user what's already done and what might be the next step. It also knows location to uncalibrated cfa-images, calibrated images, demosaiced images, registered images etc... so user can do any step again if wanted.

The project file approach is also required by my user interface.

User Interface

Eventually I'd like to have a graphical user interface (most likely PyQt) but for now only command line. I'm starting to like IRIS and its command window so that's what I had originally in mind. I got deviated a bit...

Here's how it's going to work:

AstroStack <operation> <project> <arguments>

Some examples (ones I have already implemented)

AstroStack init Andromeda

- Initialize a new project by the name of Andromeda

AstroStack adddir Andromeda light /media/.../Andromeda/

- Add all files of proper type from specified directory to project as light pictures. Actually this isn't fully implemented. It works now without the "light" argument but it asks what the type is.

AstroStack addfile Andromeda light /media/.../Andromeda/IMG_6234.CR2

- Add one file to project. Otherwise same as before. This probably should be made to understand wildcards.

AstroStack stack Andromeda flat

- Stack the flat images in the project file. This creates a file called masterflat.fits in working directory and will save the information about masterflat in the project file. I'll change all the file names to be project specific so there can be several simultaneous projects that don't interfere each other.

AstroStack demosaic Andromeda

AstroStack register Andromeda

- These are in the code but haven't been tested. Probably won't work yet.

I hope this shows the idea how the program is going to work. I'm also thinking of GUI implementation all the time and I'm trying to write everything compatible for it so no rewrite should be necessary then.

Check and rewrite everything for UI compability

I didn't really think how the UI is going to work when I first wrote this. Everything is classes and objects so it should be trivial to get everything working on UI instead of test script. Mostly the changes have been about where method gets it's arguments. Incorporating project files also requires changes so there won't be several places to update information.

Decide on image format

First I used FITS via astropy.fits, but I ran into some problems with it. I changed image format to TIFF via Pillow.Image but I've ran into even more problems... FITS might still be the better choice. AstroPy also has one advantage to Pillow: It's designed for exactly this use. I understood it supports automatically slicing large arrays into small ones which is handy for calculating medians.

Problems with TIFF

• Pillow indexes arrays differently than SExtractor does with FITS. This requires coordination system changes in registering
• Saving intermediate files with better precision than the final. Pillow doesn't like my numpy.float32's and such.
• Saving RGB images. For some reason all RGB's I tried to save were only mess.

Problems with FITS

• Creating them. I learned the right switches for Rawtran, but the problem is the program itself. I dislike the idea of having a dependency people have to compile themselves. I'll try to change this to DCRaw only or DCRaw + ImageMagick combo
• Final result should be other than FITS for easier postprocessing. TIFF suits fine, but that means Pillow would remain as a dependency

ImageMagick is being used for affine transformations and I'm not sure if it can do that to FITS files. Better try soon before deciding. Otherwise I'm choosing FITS.

Change IDE

Does concern the program a bit. Eclipse was massive and sluggish and I constantly had problems with either PyDev or eGit (or whatever the Git plugin was called). I ranted about this on IRC and got suggested PyCharm. The free version seems extremely nice, fast and git works out-of-the-box. First things I did after changing to PyCharm was to fix all the PEP8 problems the program suggested. Agreed, the code is much readable now.

I'm also trying 2-panel setup. I've been switching between to files so much that this could be useful.

Better demosaic and median stacking

I want these to be ready before releasing version 0.1. Demosaic is now bilinear and the way it lost all the colours on my test images suggests it doesn't work too well. I found something called LaRoche-Prescott demosaicing algorithm and I decided to give it a go.

Stacking works now only by mean value, which is fast and easy to implement. Eventually I want to have sigma medians and such but for the first release I want at least median stacking.

Memory usage

While running tests, I ran out of RAM. And I have 16 GB of it. So that's a problem. Seems like Pythons garbage collecting isn't that efficient without programmers help. I managed to make stacking work on about 5 GB (with 30 light images) but I still think that's too much. Have to make that lower.

That's about it...

A lot to do as I said. Where I'm now?

This can already be done, so core modules seems to work. It's everything around it that needs work.

BTW, I've been using Andromeda galaxy as my test images and hence all the images so far processed have been about Andromeda. I decided to give names of celestial objects to different versions of pyAstroStack. Version 0.1 will be Andromeda. That's already a name of a branch in BitBucket.