FISH spectral workflow tutorial

Data volume. The three FITS files total 17,182,080 bytes (16.39 MiB). All 19 beams of Sun17_A_01 would require 82,037,318,400 bytes of narrow-band SCI and 10,288,290,240 bytes of extracted CAL; the whole scan is unnecessary for this calibration example.

02 / Run it locally

Installation and execution

Use a Python environment with Python ≥3.8 and a compatible scientific stack. The normal package installation includes the full pipeline dependencies; the tutorial itself uses NumPy, SciPy, Astropy, and Matplotlib. No server connection is required. The unattended workflow needs no desktop display; the Qt viewer does.

tar -xzf fish-sun17-a01-source-20261025.tar.gz
cd fastfish-0.1.0

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[notebook,qt]'

python examples/minimal_calibration/run_workflow.py \
  --output "$HOME/fish-suna01-workflow"

cd examples/minimal_calibration
jupyter notebook walkthrough.ipynb

03 / Desktop interface

Interactive calibration with the Qt viewer

Use the downloaded Sun17 example to run the actual FISH desktop viewer on your own computer. The instructions below create a session with 32 real spectra and their measured sky positions; no connection to the original data server is required. The screenshots show the calibration and baseline pages.

1. Install and open the downloaded example

From the unpacked fastfish-0.1.0 source directory, in the same Python environment used above:

python -m pip install -e '.[qt]'
python examples/minimal_calibration/prepare_viewer.py \
  --output "$HOME/fish-suna01-viewer"
fish-viewer --session-state "$HOME/fish-suna01-viewer/fish_session.json"

Run the viewer on a machine with a desktop display; a headless Linux server needs a configured desktop/display session. The preparation command requires a new empty output directory and leaves the released data unchanged. To reopen a session and retain your edits, run only the last command. In the viewer, raw rows 0–31 refer to this compact extract and correspond to original Sun17 SCI rows 1040–1071.

2. Inspect the temperature reference

1. Select Tsys calibration in the Stage menu. Click one of the 32 points in the footprint; Previous / Next also changes the selected spectrum.
2. Keep Tpsr smoothing = 10, change ON−OFF smoothing from 10 to 5, then click Update preview. Compare the reference curves. This first filter changes the ON−OFF used to derive Tpsr.
3. Restore ON−OFF smoothing to 10, change Tpsr smoothing to 5 and update again. This second filter acts on Tpsr before matching SCI timestamps. The two changes need not have identical effects.
4. Use Reset edits to discard pending, unapplied edits. Once an edit has been applied, restore a value explicitly and click Update preview. Use values 1–10; larger filters require a longer CAL extract and are rejected in this teaching session.

3. Inspect calibrated spectra

1. Choose Apply calibration. The upper plot is raw SCI counts; the lower plot is temperature in K for the same observation. No baseline has yet been subtracted.
2. Switch between AA and BB. Compare their gain-calibrated spectra; do not expect the backend counts or bandpass to be identical.
3. Use Z for horizontal zoom, Shift+Z for both axes, B for the previous view, F for full extent, and Escape to cancel selection. For exact limits, use View → Plot ranges….
4. Open View → Display tools to switch between centered steps and lines, or adjust display averaging. These settings affect drawing, not the calibration arrays.

4. Try baseline and spectral-output previews

1. Choose Baseline. Inspect the observed spectrum, fitted components and residual. The initial list contains sin/cos plus a local degree-1 polynomial. Each component fits and applies only within its listed range; protected H I and detected RFI do not anchor the fit.
2. Use + Add component, choose a function and range, then Update preview. Change one parameter at a time. If a period reaches its search limit, open Details… and inspect the residual rather than assuming the fit is correct.
3. Choose Table. Set rebinning n_b or the Gaussian target n_g, and the output velocity limits. Update the preview. Select a region on the footprint with R (rectangle) or P (polygon), then use Region mean to inspect its mean spectrum.

What this download supports. All 32 positions, midpoint times and LSRK frequency factors are real values saved from the verified Sun17 reduction. No sky coordinates are invented. The desktop viewer and complete workflow use these LSRK factors. The separate calibration-only sub-example uses topocentric frequencies. This short, one-beam extract supports saving all four spectral stages, but is unsuitable for a meaningful spatial cube. Only gridding is disabled. The full raw telemetry and full scan are not needed to run it.

5. Save the processing stages with your chosen settings

After inspecting and applying settings in the viewer, open FISH in the same environment and restore that session:

fish --viewer qt

restore ~/fish-suna01-viewer/fish_session.json
tsyscal run
wait
applycal run
wait
baseline run
wait
table run
wait
status

Run the commands one stage at a time and inspect any failure before proceeding. These commands process the compact dataset itself. Run commands… copies this same sequence, ending at Table. If editing settings in the terminal while the viewer is open, refresh the viewer before making further edits. Different settings receive different stage fingerprints; previous results are retained.

fish --viewer qt

project /path/to/your/project
file in org_spec/your_scan/your_date/*.fits
list
go where /workers 4
tsyscal

tsyscal run
wait
applycal run
wait
baseline run
wait
table run
wait
status

04 / Follow the calculation

Calibration procedure

The example calls the production calibration functions. It does not maintain a second estimator just for the tutorial. Open each step to see its meaning and code location.

05 / RFI exclusions and baseline removal

Fit and inspect the baseline separately in AA and BB

Calibration converts backend counts to temperature; it does not remove the spectral baseline. The baseline stage fits each polarization independently, subtracts its model on native channels, and only then averages the residual polarizations.

1. Inspect RFI and H I first. In Apply calibration, zoom through the native band and compare AA/BB. Red bands mark detected RFI; gray marks protected H I. Catalogue priors are not automatic masks. The selected sample currently has no catalogue-guided detections, which does not prove that all channels are clean.
2. Choose Baseline. The component table initially has a full-band sin/cos component and a degree-1 polynomial over −1500:1500 km/s. Components fit sequential residuals in list order. Use the arrows to change order or × to remove a component.
3. Protect H I. The default protection is ±250 km/s. Protected channels and detected RFI are excluded from fitting anchors, but the model is subtracted beneath the protected line. Choose a different interval only after inspecting the actual profile.
4. Adjust one component. Choose polynomial and its degree, or sin/cos with automatic, fixed or bounded period in MHz. Set the velocity fit/apply range, then click Update preview. Each component is applied only within that range; a local model is not extrapolated outside it.
5. Inspect the residual. Check both polarizations, especially offsets and coherent structure outside H I and excluded RFI. Compare plausible models. A flat-looking average or low residual outside H I does not establish the true baseline under the protected interval.
6. Save the result. After applying settings, run baseline run and wait in the restored FISH session. If calibration settings changed too, complete the upstream reference/calibration stages first.

Inside protected ±250 km/s, the baseline is inferred from outside anchors and the chosen function. The fit alone cannot measure its true error there. Native-channel scatter also contains radiometer noise; the plotted channel means have lower scatter and should not be confused with native noise.

06 / Doppler alignment and resampling

Generate the final spectral table

The Table stage uses the actual per-spectrum LSRK frequency factors bundled with the example. It aligns baseline-subtracted spectra onto a common output grid and retains one row for each selected timestamp. This is a spectral table, not a spatially gridded cube.

1. Choose Table in Stage. Keep rebin mode with n_b=2 and output velocity limits −250 to +250 km/s for the default example.
2. To compare responses, change the rebin factor or select Gaussian with a target n_g in native-channel FWHM units. Update preview and inspect the result; display averaging is a separate plotting control.
3. Optionally select points with R (rectangle) or P (polygon) on the footprint and use Region mean. This previews a mean without spatial gridding.
4. Run table run, then wait and status in FISH. The default saved table has 32 rows × 2,485 channels. There is no need to run xy_map for this example.

from pathlib import Path
import json
import numpy as np
root = Path.home() / "fish-suna01-workflow"
manifest = json.loads((root / "table_manifest.json").read_text())
processed = json.loads(Path(manifest["processed"]).read_text())
product = processed[0]["products"][0]
spectra = np.load(product["data"], mmap_mode="r")
frequency_ghz = np.asarray(manifest["frequency_ghz"])
print(spectra.shape)  # (32, 2485) for the defaults

07 / Check your answer

Saved outputs and verification

The compact inputs were compared with three surrounding original CAL files and the full original Tcal file. At the default smoothing settings, the effective Tpsr and all calibrated SCI values were exactly equal on the server (maximum difference 0 K). The separate calibration-only script and notebook additionally compare selected native temperatures and all gain coefficients with frozen original-input results. Small floating-point differences across platforms are allowed.

The unattended runner also saves calibration.png, baseline.png and table-spectrum.png. All four stages process 32 spectra. The default table is fully finite; RFI-fraction fields remain within [0,1]. The workflow does not run spatial gridding.

Reference quality. A few nonpositive Tpsr values occur at 1420.28808594 MHz (about 0.084% of AA and 0.048% of BB reference samples). That channel is inside the excluded H I gain interval and is not used as a gain anchor. These values are reported, not changed or automatically labelled RFI.

Real-data repeatability does not establish the true astronomical spectrum or absolute calibration accuracy. The calibration diagnostic plots native frequency as topocentric velocity. The baseline plots, viewer and final spectral table use the bundled real LSRK factors. Baseline and RFI quality still require scientific assessment.

08 / Parameter changes

Parameter tests and code modification

python examples/minimal_calibration/calibrate.py \
  --output "$HOME/fish-calibration-no-smoothing" \
  --n-t-cal 1 --n-t-apply 1