Setup

Get it running locally on your machine

Warning

pylim is now a stand alone package and can be installed from GitHub. This section is outdated with respect to pylim.

For now (November 2021) the easiest way to use this is to fork this project and install pylim in your python environment. If you want to work with the analysis and quicklook scripts you can just work in the cloned repository and add your files as you wish. A possible structure could look like this:

├── phd_base  # source code managed by Johannes`s github account
│   ├── analysis
│   ├── docs
│   ├── processing
│   ├── src
│   │   └──  pylim  # actual python module
│   ├── quicklooks
│   ├── your_folder  # source code managed by your git
│   ├── config.toml
│   ├── LICENSE
│   ├── README.md
│   ├── requirements.txt

Installing pylim

pylim is a full module complete with a pyproject.toml and a setup.cfg. If you only want to work with the functions provided by pylim and don’t care for the other scripts too much, the easiest way of installing pylim is to download the .tar.gz file from the dist folder. The .tar.gz file holds the complete package and can be installed like the self build distribution (see Windows10/Unix - (Build distribution and) install from it).

If you forked/copied the whole repository you can also install it from source or build a distribution and then install it. All ways work but the first one should be the easiest. These are the ways it works for me on Windows10 and Unix.

Windows10/Unix - Install from source

Open an anaconda prompt (Windows10) or your bash shell (Unix), move to your project directory (e.g. phd_base) and activate your working environment.

cd C:\Users\USERNAME\PyCharmProjects\phd_base
conda activate phd_base

Then install pylim from source with:

python -m pip install .

Windows10/Unix - (Build distribution and) install from it

If you already have the tar.gz file downloaded you do not need to build the distribution anymore. Just skip to the installation. Otherwise start here:

Make sure you have the newest version of build installed:

conda install build
conda update build

From your project directory (e.g. phd_base) you can then call:

python -m build

You will see a lot of output and hopefully this line:

Successfully built pylim-|version|.tar.gz and pylim-|version|-py3-none-any.whl

For more information see Generating distribution archives.

You will now have a new dist folder in your project directory with the two mentioned files above. To install pylim from your anaconda prompt call:

python -m pip install .\dist\pylim-|version|.tar.gz

Using the .tar.gz file will delete any old installations of pylim. Whatever way you chose, you should be able to import pylim now:

import pylim.helpers as h

Data structure

HALO campaign data is organized by flight in the 01_Flights folder, so that every flight has its own folder with subfolders for each instrument in it:

├── 01_Flights
│   ├── all
│   │   ├── BACARDI
│   │   ├── BAHAMAS
│   │   └── horidata
│   ├── Flight_20210624a
│   │   ├── BACARDI
│   │   ├── BAHAMAS
│   │   └── libRadtran
│   ├── Flight_20210625a
│   │   ├── BACARDI
│   │   ├── BAHAMAS
│   │   ├── horidata
│   │   ├── libRadtran
│   │   ├── quicklooks
│   │   └── SMART
...

In order to be able to work across all flights an additional folder can be found called all. This folder contains one folder for each instrument which holds all data for the whole campaign.

This data is stored on the server but can also be stored locally. To access it without needing to worry about changing the paths every time one switches from the server to local data, the function pylim.helpers.get_path() is used together with config.toml to generate the correct paths. In the configuration toml file the path to each instrument can be defined either as a absolute path or -to allow for easy path creation- relative to the base directory and the flight folder. Providing pylim.helpers.get_path() with the instrument key (e.g. “smart”), the flight (e.g. “Flight_20210625a”) and the campaign (e.g. “cirrus-hl”) the correct path will then be created according to the current working directory. Providing the key “all” requires the additional keyword instrument and will get the path to the instrument folder in the all folder.

Attention

The config.toml file has to be in the current working directory of the python console. So when you run a script in a different folder (like processing) be sure to copy your most recent config.toml to that folder as well. Or you change into the directory with the config.toml using os.chdir().

There are two ways of setting up paths to your local data source:

  1. Edit the existing paths under jr_local or jr_ubuntu depending on whether you are using Windows or Linux.

  2. Create a new campaign which defines the paths as you need them.

The second options is kind of hacky but would allow everyone to use the same config file. However, I don’t see a merit in that so number 1 would be the preferred option. Every user should have their own toml file.