By Jenny Bailey
last modified - 01/05/01 22:50
Version 1.02 now released
For all those that don't know what a version 1.0 is like, remember that we now need feedback for all the problems that you find.
Announcements in the SETI League website will indicate when further released versions are available.
There are many FFT programs available, and some designed specifically for SETI use. I have started Setisearch with specific reasons in mind :
Open source : much of the code should be re-usable under any O/S. This can be used as a library for anyone else's project.
Modularized : the program can easily be modified for other input - such as external DSP card
Separate display : my plans to use a remote telescope for piggy-back SETI, the data processing can be done close to the telescope, and the display can be seen over an internet connection
Language : ok, so I should have used c++, however I am more familiar with c and I wanted to finish this sooner rather than later. Further to this, I have been persuaded to compile the program with g++ ( GNU c++ ) because the warning messages are more pedantic.
This program is not another SETIFOX - an excellent SETI program written by Dan Fox for SETI for use with DOS or Windows. However, I dream that Setisearch will be the equivalent Linux package in a couple of years.
Initially I hope that Linux enthusiasts will overcome the hurdles to get this program installed and working ( this is not for the faint-hearted ). As the program matures - it will become more user friendly and distributed as an rpm ( or .deb or whatever - we don't go in for distribution wars here ).
There is a lot of Techie ability and enthusiasm in the Linux community and drawing some of this towards the SETI community would be no bad thing.
The functionality will become available in Phases.
Supports SB16 compatible in mono mode using Linux /dev/dsp interface ( Linux Kernel drivers invisibly support many soundcards )
AGC of input signal using Soundcard's mixer level control
'AGC' level logging for Radio Astronomy purposes
Multiple buffer arrangement to reduce data loss when windowing
Calculate target position ( RA and Elevation ) in real time
Normalize mode to flatten receiver audio path response
Averaging and Peak detection
Logging of 'Hits'
Separate display program connects to main program via socket ( could be over network/ internet )
documentation
Frequency scanning to IC-R7000, FX5000 and any other radios that I can get hold of
Stereo
Hard programmed response on signal detection - Alarms, aerial movement - all by waggling parallel port
distribution
simple programming interface to add new radio protocols ( for scanning )
simple customizable actions on receipt of signal - move aerial, frequency, send email, fax etc.
support more than one soundcard
Interface to sharc development kit - for more bandwidth
A block diagram, broken into the relevant threads.
Audio is taken from the /dev/dsp device. This is the soundcard's A/D converter and it is programmed to supply a stream of 16 bit words. The hardware and data management of /dev/dsp is taken care of by the Linux Kernel.
The first problem was that this device will supply data at the sample rate ( default 8000 samples/second ), and then once the program wanders off to process these results, any further data from this device is lost for the duration of the calculation. It was therefore decided that the program should be multithreaded. The threads are :
Reading Thread : Data is taken in from /dev/dsp in a block size of fftlength/2. This thread blocks until the required length of data is received. This buffer is then marked as full.
Processing Thread : Data is taken from buffers and FFT'ed, then averaged. The resultant data is put into a result buffer.
Telescope Position Thread : Calculates the sidereal time every 2 minutes for logging, and hit detection
Frequency scanning Thread : Writes a new frequency to a radio every < scan interval >
Power Log Thread : Writes average audio power to a logfile every < log interval >
Setisearch is designed to FFT incoming audio as fast as possible, hopefully a 2048bit FFT at 44KHz, stereo. All the moving data between buffers must take valuable time. The same goes for the averaging. Perhaps we need a better architecture.
Consider the following diagram :
For the above example we will use an FFT length of 1024 bytes. As each fft is performed on two samples ( to recover some of the data loss due to windowing ), the sample size is 512. The soundcard is in 16bit mode, so each buffer consists of words.
An FFT and Bit reverse is performed on the target buffer. This buffer consists of both real and imaginary parts ( each stored as float ). Real magnitude values of the sampled waveform are stored in the real part of the array, imaginary parts are set to zero.
After the FFT and bit reverse, the bin level consists of both real and imaginary parts. The magnitude is the square root of the sum of the real ( squared ) and imaginary ( squared ) parts. The phase information can be discarded ( unless anyone can think of a good use for it ).
We don't bother with the squared root, as this is unnecessary because we are looking for relative peaks. A number of these results are added into the average buffer, and then divided by the average number ( probably unnecessary again ).
The normalizing process is designed to take a standard receiver response ( peaking at 1KHz, tailing off to each side because of receiver filtering ) and flatten it where possible, and removing response where there is not enough level. This will greatly help the peak detection.
Normalizing involves running Setisearch for a few minutes ( on the target receiver with no signal ) to get a nice noise response. This response is then saved in a file. The contents of this file are then used to 'flatten' the receiver response.
Audio bandwidth from receiver = 20KHz ( DC .. 20KHz )
Sample rate ( SoundBlaster 16 ) 44HKz
Data rate ( assuming 16 bit samples ) = 88Kbytes/second
Target bin size = 10Hz
Bin Size = sample frequency/fft length = 44000/4096 = 10.7Hz
Number of ffts per second = sample frequency/fft length = 10.7 ffts
per second.
Max time for each fft cycle = 93ms
|
Level |
Function |
|
0 |
No output ( you probably don't want this ) |
|
1 |
default - severe errors only |
|
2 |
non fatal errors - program can continue |
|
3 |
interesting errors |
|
4 |
not assigned |
|
5 |
all above + debug level 1 ( setup reported ) |
|
6 |
all above + debug level 2 ( some processing reported ) |
|
7 |
all above + debug level 3 ( terse buffer management ) |
|
8 |
all above + debug level 4 ( normal buffer management ) |
|
9 |
all above + debug level 5 ( verbose buffer management ) |
|
10 + |
insane levels of debug, you can't expect the program to run properly. Development only. |
The error report will be in the form of :
n:function:error
where n is the level of the error or debug
ie
Error 3::sidtime:Incorrect Planet - this only works on EARTH !
At startup, Setisearch will look for a file called setisearch.cfg ( case { capitalization } is significant ). The file is a straight text file, customizable with vi, emacs etc. It contains one line per entry, comprising a keyword and a value.
|
Parameter |
Default value |
Description |
|
fftlength |
1024 |
Number of samples to be FFT'ed |
|
locationEW |
0.00153 |
Location of SETI station in degrees from the Greenwich Meridian. East is positive. |
|
locationNS |
52.2203 |
Latitude of SETI station in degrees from the equator. North is positive |
|
altitude |
30 |
Aerial targeting above the horizon in degrees. This should still be positive in the southern hemisphere. |
|
azimuth |
180 |
Aerial targeting in degrees. Zero is North |
|
testMode |
1 |
0 = no test mode, ie get audio from the soundcard 1 = test mode, use a synthesized signal instead of the soundcard |
|
sampleRate |
8000 |
number of samples to be collected per second |
|
startFrequency |
1420.000 |
Start frequency, and the Frequency the scan resets to on each loop ( MHz ) |
|
frequencyIncrement |
0.025 |
Frequency Increment in MHz - Its probably sensible to make this slightly less than your fft bandwidth ( fft bw = samplerate/2 ) |
|
maxFrequency |
1421.000 |
Stop frequency after which scan loops back to start frequency |
|
frequencyPauseTime |
20 |
Time ( in seconds ) that we sample each frequency |
|
fftWindowFunction |
2 |
|
|
averageCount |
200 |
Number of samples that are averaged. |
|
pwrLogTime |
120 |
pwrLogTime is how often ( in seconds ) the incoming audio power is recorded to disk |
|
alarmThreshold |
75 |
alarmThreshold - assume that the average signal per 'normalised' bin is 50%, then we look for any signal above that a system set up to be 'sensitive' would be 60%, 'insensitive' would be 200%. It depends how load your alarm is.. |
|
alarmTimeout |
120 |
how long an alarm remains active after the signal goes - in seconds |
Any line that is blank or beginning with a # is ignored
The order of the entries is not important
There are to be a number of other parameters added.
frequency limits, pause times and step sizes
radio type for frequency programming format
mono/stereo/multiple sound cards
The normalization file corrects for the typical receiver response as shown below.
The amplitude ( y-axis ) is in arbitrary units ( squared ) and the frequency ( x-axis ) is from DC to 22KHz. The shape of the response is due to the RX IF filter and audio processing components. The receiver is a modified FR5000.
After normalization the response is as shown :
( Note that this is the setidisplay.tcl program - used mostly for debug mode )
The normalisation.dat will contain one entry per frequency bin. Each entry consists of a bin frequency and a level from 0..1. This level, when multiplied by the smoothed bin level for a particular receiver, will give an equal level in each bin. This is a bit experimental. It should make the display more pleasing, but more importantly help peak detection. Where the receiver response is below a certain level ( near to DC and above the pass band ) the entry will be zero.
A normalization file will have to be re-created each time you change receiver, or FFT length. Yes, I know, yeuch. But you don't change your receiver that often.
The source is in .tar gnu zipped format. Tar files are common under Unix, and Linux supports a gnu version of zip.
To extract under Unix, use the command :
tar -zxvf setisearch.tgz
The binary setisearch has been built under RedHat 6.2, but can be compiled for most Linux distros
|
Version 1.01 |
Download |
|
Version 1.02 |
Download |
To build Setisearch, use the commands :
make clean
make
Winzip 6.3 ( Under windows ) will extract .tgz files, although you will need a Linux PC to build and run the program. The .cpp source is text readable ( as is my programming style - I hope ).
I intend to make the program available as a .rpm package. This is a 'RedHat Package Management' file that allows easy installation. Many Linux distributions are adopting this format.
Configure your soundcard using the Linux app 'sndconfig'
Set up mixer input level using something like xmixer ( I know that setisearch should do this )
Download the setisearch rpm file from here
install as root :
rpm -i setisearch-1.0-1.i386.rpm
run setisearch with a debug level of 6 ( you might need to set permissions to access /dev/dsp and /dev/mixer )
setisearch 6
wait for 2 minutes, then assuming you have no incoming signal press 'n' to normalize the receiver response
Setisearch is a text-only program. Processing power to drive a display should be used for analyzing data.
So far, Setisearch is in the following state :
read data from /dev/dsp 100%
AGC 60%
windowing 90%
FFT 100%
normalize 100%
peak detect 40%
display 80%
telescope position 90%
frequency scan 90%
alarm handling 90%
The display program is used for Setisearch development at the moment. It draws a basic line graph of the magnitude in each frequency bin. A waterfall display using the same socket interface will be a future development.
This is a TCL program using the BLT extensions to TCL. It draws good graphs.
The current setidisplay program draws a basic graph of the magnitude of each frequency bin. This is very basic. I would prefer a waterfall display. I am, however, conscious that the prettier we make the display, the more CPU time is uses when we should be using that to look for signals. This is at least some of the reason that setidisplay should be used from another PC over a network
Setidisplay is currently distributed as setidisplay.tcl. You will need the BLT TCL module to run this. You can download BLT from here
I intend to compile setidisplay.tcl into a binary ( at least I hope that I can do this )
There is also a basic 2-D waterfall display - again in development - I would like this to be a 3-D one much like the seti@home display.
Source for both modules Updated 03-09-2k
Waterfall tcl program
This program takes in the hit log and reproduces a picture of the signal as follows:
Download it here
The hit files tend to be 9 - 60MBytes long depending on the number of hits for that day, and the duration of the hit.
Even using bzip2, my shortest log file only compressed to 1.6MBytes.
The position fields will be completed once the data file contents are valid.
Change Log
|
Date |
Change |
|
11-05-2k |
Added socket support for display program ( this breaks when telnet session is quit ) |
|
12-05-2k |
Added display program - setidisplay.tcl |
|
14-05-2k |
setidisplay.tcl now updates each time setisearch puts data out ( 10s ) |
|
14-05-2k |
setisearch - modified soundcard DMA programming - not yet perfect |
|
24-05-2k |
whilst working in Peru, I've changed the output format to the socket changed the buffer management added some averaging updated this document |
|
09-05-2k |
Added Frequency scanning thread Added a windowing function ( I'd like to play genetic programming/sensitivity with this ) Improved the averaging algorithm - more work required here Added more configuration file options - window type and average length |
|
12-6-2k |
Changed Frequency scanning parameters from floats to doubles Changed the debugging output to show debug level |
|
14-6-2k |
Added comment lines into setisearch.cfg file |
|
18-6-2k |
Added parameters for logging at peak detection |
|
02-07-2k |
Updated this page Added logfile writing thread ( same format as specgram ) Made error/debug reporting more consistent Noticed that I get buffer overruns at 44KHz/4096 on PII/233. top reports that I am only using 10-20% CPU time. This needs investigating. Setisearch is intended to optimize your processing time Mixer programming seems to work, AGC control loop to be created. Using real radio, normalization file seems to work to some extent. I think it might be level sensitive. I'll try it again when I have AGC working. I realize that I need to start using RCS and version numbers. setisearch.h tidied up |
|
29-07-2k - 06-08-2k |
Improve normalization file reading and writing - commands 'n' and 'r' now work from inside setisearch Alarm function now detects peak - put delay onto alarm detection thread until exponential smoothing settle ( 60 seconds ) add smoothing coefficients into setisearch.cfg write waterfall display - waterfall.tcl |
|
08-08-2k |
put files into RCS ( at last ) |
|
17-08-2k |
change makefile to add -Wall, -g and -pg options |
|
18-08-2k |
Change normalisation function so that it works. It was fighting the exponetial smoothing function. I have also moved all the normalisation routines to normalise.cpp |
|
05-10-2k |
Created RPM of setisearch 1.0. Tried this on a virgin RedHat 7.0 installation. I documented the problems ( soundcard, mixer ) Created a hit analyser program - hit_analyser.tcl added better reporting of hit direction into hit files |
I would like to give credit to the following people so far
Matthew - Inspiration, code reviews and understanding
Nicky - Making me use c++. Also, the power display program. ( not on here yet )
Phil Karn's FFT code
Jennifer - programming advice
The Lib Dems who offered Threads consultancy whilst conducting an election downstairs
seti-uk - SETI-UK web page
Hardware - Our home seti station