- Email: [email protected]
Mariner 1969 television image processing
Pattern Recognition
PergamonPress 1970. Vol. 2, pp. 261-268. Printed in Great Britain
Mariner 1969 Television Image Processing J A M E S A. D U N N E Jet Propulsion Laboratory, 4800 Oak Grove Drive, Pasadena, California 91103, U.S.A.
(Received 24 February 1970) Abstract--Mariners 6 and 7 sent back approximately two hundred photographs of Mars, taken from distances ranging between one million and three thousand km relative to the planet surface. These images contain true object space information distorted, to one extent or another, by the signature of the imaging system itself. In order to maximize the scientific yield of the television data, the natures of the distortions introduced by the imaging system must be quantitatively determined, and techniques developed whereby they can be accurately removed. Television systems typically exhibit photometric nonuniformities and nonlinearities, geometric distortions, both optical and electronic, image retention and spatial frequency response limitations. In addition, Mariner 1969 specific problems were introduced in the course of on-board processing applied in the interest of data compression and etficient bandwidth utilization. A cubing amplifier and an automatic gain control circuit were employed, with photometric "DC reference" supplied in the form of a separately encoded, under sampled raw signal. The effects of on-board processing can in principle be removed by means of combining these separate data streams. Digital image processing techniques have been developed at the Jet Propulsion Laboratory which enable us to characterize these various distortions using calibration data, and to remove them from images returned by the two Mariner spacecraft. This paper will provide brief descriptions of some of these techniques and illustrations of their application to Mariner 1969 image data.
INTRODUCTION THIS paper will describe briefly the nature of the Mariner 1969 TV system and imagery, and some of the processing procedures being applied to the returned data in the Image Processing Laboratory (IPL) of the Jet Propulsion Laboratory's Space Sciences Division, a facility consisting of a dedicated IBM Model 360/44 computer and special purpose peripheral equipment, image scanning and display hardware and a software system designed for image manipulation and processing. The IPL is staffed with personnel specializing in image analysis, image processing technique development, hardware development and operational support. The major phases of the IPL support to the Mariner 1969 television experiment included preflight video subsystem calibration support, near-real-time flight image analysis for mission operations, and detailed post mission television image processing in support of the television experiment team. Preflight calibration activities included participation in the design and preparation of special procedures and test targets, preliminary calibration data analysis for engineering evaluation, and the detailed reduction of calibration data for the quantitative restoration of flight imagery. Inherent in the preflight activities was the development of image processing techniques and software for the correction of Mariner '69 unique video subsystem characteristics. The initial post mission image processing phase is, from the viewpoint of science data analysis, preliminary, representing as it does a prerequisite to the quantitative analysis of the Mariner TV data. Because of the non-linear contrast emphasis electronics inherent in the television camera system, digital processing is required before even approximately 261
~.
,
3,o
,~
s~
-
t
i
+++
F
t
i 4
-
. . . .
~
'
',LINE
++++l BY I LINE
1 ----~
:
EVERY7TH PIXEL, A/D TO 8 lilTS, 6 LEAST SIGNIFICANT BITS DIGITAL RECORD, 16.2 KBPS
ON EARTH
PIXEL = PICTURE ELEMENT M IV 2(]0 PIXELS BY 200 LINES
++* * DIGITAL DATA FROM OTHER EXPERIMENTS PLACED ON DIGITAL RECORDER
....
EVERY 28TH PIXEL, ,6./1:), 8 BITS, 4 CENTER BITS, DIGITAL ENCODE, PLACE ON ANALOG RECORDERDURING RETRACE TIME
SPACECRAFF
DIGITAL VIDEO
EVERY 28TM
TELEVISION DATA FLOW
FIG. 1. Mariner 1969 TV data flow.
TYPICAL LINE OF VIDEO
RETRACE TIME FREQ. 18.9 KHz
( 60 MSEC/LINE) (704 LINES/PICTURE ) = 42.24 SEC/PICTURE
i
. . . . . .
945 PIXELS TELEVISION FRAME FORMAT
,
~EC
.......... 945 PIXELS/50MSEC, AGC, PREEMPHASIS, ANALOG RECORD, 18.9 KHz CARRIER
j,
-°
,,,,i
z
~o
............ ~............ i ........... J.......... ~........... ---',
i,o
ACTIVE LINE TIME
LINE READOUT TIME: 60 MSEC
DSS
DATA STORAGE SUBSYSTEM
ANALOG VIDEO
z=
Z
Mariner 1969televisionimageprocessing
263
quantitative intensity measurements can be made on the returned pictures. The full realization of the scientific objectives of the television experiment requires additionally careful correction for systematic noises, photometric non-uniformities and non-linearities, geometric distortions, and modulation transfer function degradation, all of which constitute the signature of the television subsystem. The series of procedures designed to remove this signature from the images returned by Mariners 6 and 7, referred to collectively as decalibration and cleanup, will be discussed herein. MARINER '69 IMAGE DATA FORMATS A simplified data flow diagram of the Mariner 1969 television system is given in Fig. i. Major subsystems are the TV subsystem (TVS), including the cameras, head electronics, video amplifiers, cuber, automatic gain control (AGC), automatic aperture control (AAC) and the data storage subsystem, including the analog tape recorder (ATR), analog channel A/D converter and digital tape recorder (DTR). Analog video is recorded in the analog mode on the ATR and digitized to six bits on playback, whereas digital video is digitized before recording on the DTR or transmission back to Earth in real time. Composite analog video (CAV) passes through an X 3 amplifier (cuber) for contrast enhancement, before being recorded on the ATR. Near encounter CAV, in addition to being acted upon by the cuber, is subject to two separate electronic gain adjustments. First, the automatic aperture control (AAC) sets one of eight gain states, depending on the signal received from the AAC sense area in a previous A camera frame. Second, an automatic gain control (AGC) amplifier, acting in real time, continuously adjusts gain to keep the CAV signal on the most sensitive portion of the cuber transfer curve. AAC gain state information is transmitted in the science data stream, but no information is available on the state of the AGC, and therefore, near encounter CAV contains no photometric DC reference. In the far encounter mode, the automatic aperture (AAC) control is locked in its lowest gain state, and the automatic gain control (AGC) is inhibited. Every seventh sample of the video data is digitized directly by a separate A/D converter in the television subsystem and encoded to eight bits, of which the six least significant bits (LSB) are either recorded on the digital tape recorder (DTR) and transmitted or transmitted directly in real time. In addition, every twenty-eighth (/28) sample is recorded on the ATR embedded in the CAV during the television retrace time. The/28 words contain only the four central bits, i.e. two LSB and two MSB are truncated. Every seventh digital video (DV) frames also contain science telemetry introduced during the active line scan producing a "data bar" corresponding to the central 26 samples of each DV line. Therefore, of 134 possible DV samples per line, only 108 are transmitted. Near encounter DV is affected by the AAC gain state but not by the AGC amplifier and can thus be used as a reference for photometric restoration. Mariner 6 returned 50 Far Encounter CAV pictures and 25 Near Encounter CAV pictures. Mariner 7 returned 93 and 35 far and near encounter CAV pictures, respectively. Of the 143 total F.E. CAV pictures, corresponding DV data is available for approximately 10. All of the N.E. pictures were, of course, received in the form of CAV and DV. In addition, some 3100 real time F.E. DV pictures were transmitted by the two spacecraft. CALIBRATION DATA The goal of the television system calibrations was the acquisition of data by means of which system performance parameters can be quantitatively characterized. Data of this
264
JAMESA. DUNNE
type is useful in two areas. First of all, they provide a basis for the selection of flight subsystems and the prediction of their response during the various phases of mission operations. Secondly, they are necessary for the development and application of image processing techniques for the quantitative restoration of image intensity and geometry information. The general categories of system performance for which calibration data must be collected include noises and photometric, geometric and modulation transfer function distortions. In each of these principal categories, data is required to characterize a number of independently varying parameters. Photometric distortions, for example, include spatial sensitivity functions for both primary and residual images, veiling glare introduced by light scattering in the optical system, shutter leaks and various electronic transfer functions and gain states. TVS calibration data was obtained on the bench, in the thermal vacuum chamber and on the spacecraft, the latter in the Spacecraft Assembly Facility (SAF) and at the Eastern Test Range (ETR). Calibration tests consisted of near and far encounter mode flat fields, geometric distortion and modulation transfer function targets, complex grey scale targets viewed in various orientations to provide residual image data, simulated lunar limb and low contrast surface scenes, orbit determination test targets, narrow bandpass filter targets for spectral response characterization and miscellaneous special tests designed to provide data on light leaks, out of field scattering, and the like. Nearly six thousand calibration frames were recorded in all, including proof test model and flight spare system calibration. SUMMARY OF DECALIBRATION PROCESSING PROCEDURES Because of the unique design of the Mariner 1969 television system, photometric decalibration must await the completion of the reconstruction process, which provides an output picture in which the effects of the on-board processing electronics have been removed. This process will be discussed in some detail in a succeeding section. Reconstructed photographs contain all of the photometric distortions characteristic to vidicon images. The first of these to be removed in the decalibration procedure is that of residual images. Calibration data provides a measure of retentivity as a function of previous exposure and position in the image format for each of the flight cameras. This information permits the construction of a residual image sensitivity map for each of the cameras, which can be used to modulate the previous picture, the resultant then being subtracted from the photograph containing the residual image to effect the desired correction. Far Encounter pictures contain residual images formed by the immediately previous exposure. However, only widely spaced F.E. pictures were recorded in order that the limited (33 pictures per tape load) capacity of the ATR could be efficiently utilized. The technique for residual image removal for Far Encounter, therefore, involves the use of subtracting the appropriately scaled portion of the picture from itself, after first translating it to coincide with the location of the residual image. The registration of the real to residual image can be accurately checked using difference picture techniques. Since the planet revolves only about ~° per frame time, and the residual image percentage is less than 12 per cent of the previous exposure throughout the image format, photometric errors introduced in this procedure are small. Photometric decalibration also involves the removal of coherent noises superimposed on the video signal, a procedure which will be described in a later section. After noise and residual image removal, correction must be made for sensitivity variations across the scanned area of the vidicon target. Calibration data provides a light
Mariner 1969televisionimageprocessing
265
transfer characteristic function (data number out vs. luminance in) for each point in the image format. These data are obtained using luminance sources specially constructed to be uniform to within a few tenths of one per cent over the entire region viewed by the vidicon target. Spatial variations in output intensity can therefore be accurately described in terms of sensitivity variations on the vidicon surface. Calibration fiat field pictures, properly scaled for temperature coefficients and spectral response, provide the basis for the final sensitometric correction of the flight data. The removal of geometric distortions from the flight data is required in order to allow the accurate determination of spatial relationships and scaling in object space. These types of measurements are important in attempts to construct maps of the areas photographed for the guidance of future exploration efforts, and for the determination of the figure of the planet in using Far Encounter photography. Two types of distortions exist in the data. There is optical distortion present in the image on the vidicon target, as in the case of any camera system. A more significant distortion, termed "electronic", is introduced in the course of scanning the target to produce the video signal. These effects can be separated and individually corrected, by means of in situ measurements of a reseau pattern which was accurately deposited on the vidicon target, combined with calibration data taken using a test target produced from grating engine-scribed glass plate. The procedure followed in Mariner '69 geometric correction involves first the normalization of the flight data, based on reseau locations determined for each individual picture, to the same target-space coordinates as a selected geometric distortion calibration frame. A second geometric correction then is conducted using parameters which have been previously determined to correct the image of geometric distortion test target to its independently measured true shape. Another correction which is made to the data involves the restoration of high spatial frequency information by means of convolution filtering. The filters, referred to as sine wave response filters in recognition of the generally sinusoidal intensity distribution of the scanning beam, are derived from calibration measurements of the modulation transfer function of the camera systems obtained using computer generated variable frequency low contrast sinusoidal targets. This brief summary of decalibration procedures touches only the major classes of processing steps, and is intended only to serve as a background for the ensuing discussion of the two procedures which dominate the Mariner 1969 decalibration processing at the Image Processing Laboratory ; reconstruction and coherent noise removal. RECONSTRUCTION As described in earlier sections, the fully sampled Near Encounter data requires the restoration ofa DC photometric reference before measurements of relative scene luminances can be made. The data formats obtained from the spacecraft are illustrated in Fig. 2. Taken counterclockwise beginning in the upper right hand corner, these are every twentyeighth (/28) digital video, every seventh (DV) digital video and composite analog video (CAV). The picture in the lower right hand corner of Fig. 2 represents the product obtained in the reconstruction process. The mottled appearance of the DV and/28 pictures results from the truncation of the two most significant bits (MSB) before transmission. The first and most difficult step in the reconstruction process involves the restoration of these bits. MSB restoration is an iterative operation, wherein automatic computations are interleaved with manual adjustments, until a satisfactory product is obtained. The primary tool in
266
JAMESA. DUNNE
accomplishing MSB restoration is a program which, in the course of several passes through the picture, attempts to restore MSB's, using criteria which essentially demand smooth intensity transitions in the output picture. Since there are two missing MSB's, the true intensity at any given point in the picture can be in any of four "quadrants" i.e. 00, 01, 10 or 11. The MSB program, then, first scans the columns in the picture looking for sharp discontinuities, which represent probable quadrant changes. It also looks for lesser discontinuities, termed "ambiguities" at which a quadrant change may or may not have occurred. The program then selects a column which exhibits the largest number of quadrant changes and the smallest number of ambiguities, and begins to correlate across rows in the picture, again using the smoothness criterion to make most probable MSB assignments. The output picture so produced is then examined by an analyst, errors in MSB assignments noted, and point-by-point adjustments made to correct MSB assignment errors. This procedure often takes several iterations, particularly in the case of pictures wherein noise bursts have produced anomalous intensity values which perturb the MSB program. Obviously, also, certain true image characteristics (i.e. small, sharp crater shadows) can violate smoothness criteria and therefore require manual adjustment of the automatic MSB assignments to produce a true representation of the scene. This, of course, implies the use of the CAV pictures as a guide in the generation of correctly restored DV. An example of MSB restoration as applied to Mariner 7 N.E. picture No. 20 is given in Fig. 3. Once MSB restoration is complete for both DV and/28, these pictures are combined with the CAV to produce the reconstructed picture. The technique used here is one of multiplicative adjustment of the CAV intensity to match that of the MSB restored digital value at the corresponding point in the picture. The gain factor used to make this adjustment is then applied to adjacent CAV points, after correction by linear interpolation to vary smoothly between digital video sampled points. The product of this process is, once again, iterated upon by a combination of hand adjustments and discontinuity-recognizing programs, to produce the final reconstructed picture. Iterations are required because of errors in either CAV or DV introduced by noise and/or phase shifts. Figures 4 and 5 illustrate the effects of the reconstruction process. They are mosaics of five Mariner 7 wide angle camera pictures of the Martian south polar cap, Fig. 4 being raw CAV, and Fig. 5 the reconstructed versions of these same photographs. Note the photometric distortions in Fig. 4, the luminance of the desert on the right appearing to be nearly equivalent to that of the "snow" mantled polar cap region. The true luminance relationships are shown in Fig. 5. Note also the dark collar around the cap-desert interface. This dark band is illustrative of the time constant of the AGC. The actual position of the sunset termthator shows up, on the other hand, more clearly at the left edge of the raw mosaic than in the reconstructed one. These pictures are also illustrative of the value of the on-board processing electronics in enhancing visible detail in the raw pictures. The true contrast range on Mars (Fig. 5) is obviously too great to allow the optimum visibility of surface detail over the whole of the region covered by these mosaics in a single photographic print. Finally, it should be noted that the residual images of earlier limb pictures visible in Fig. 4 can only be removed after the completion of the reconstruction procedure since this process relies on a knowledge of the true exposure level in the picture which produced the residual image. COHERENT NOISE REMOVAL Mariner '69 data is characterized by the presence of a significant amount of structured noise introduced into the video signal by pickup from power supplies, heater wires and the
I~[t~l.ff-iM l Y l ~ , t . ~ $ P t , / I I ~
,
FxG. 2. Data formats. Clockwise from lower left, composite analog video, /7 and /28 digital video and reconstruc~,ed picture.
[facing p. 266
FIG. 3. MSB restoration for /7 digital video frame.
FIG, 4. Raw CAV mosaic of the Mariner 7 polar cap sequence.
FIG. 5. Mosaic of reconstructed versions of the Mariner 7 wide angle camera polar cap sequence.
i
08-31-69 110410 JPL/IPL
09-04-69 085843 JPL/IPL
FIG. 6. Mariner 6 Near Encounter picture number 18 and its two-dimensional Fourier transform.
PICTURE COUMT 6, S/C 63, STATIOM 21, CAMERA ID DAY 213, GMT 02195139 STRETCH
PICTURE COUNT 6, S/C 63, STATIOM 21, CAMERA ID i DAY 2±3, GMT 02195139 STRETCH 518X512 FOURIER TRAMSFORM FFT2 FFTPIC LINEAR AMPLITUDE/AMPLITUDE
7M19 DAY 217 04.53.04 A CAMERA GRM2 GAIM 1.0 PHASE AMGLE 35 SDLAR ZEMITH AMGLE 76 APPROX. WIDTH iSZi KM HEIGHT 1294 KM RIZEAU SAR FFT2 FFTPIC LIMEAR CPIC AMPLITUDE 12-06-69 180614 JPL/IPL FIG. 7. Two-dimensional Fourier t r a n s ~ r m of Mariner 7 Near Encounter picture number 19.
7N19 DAY 217 04.53.04 A CAMERA GRN2 GAIN 1.0 PHA3E ANGLE 35 SOLAR ZENITH ANGLE 76 APPROX. WIDTH 1821 KM HEIGHT1294 KM RIZEAU SAR SAR ~AR DROPOUT NORING NOISREM DIFFPIC FFT2 FFTPIC LIMEAR CPIC AMPLITUDE 12-09-69 111739 JPL/IPL FIG. 8. Two-dimensional Fourier transform of Mariner 7 Near Encounter picture
number 19 after partial removal of coherent noises.
7M19 DAY 217 04.53.04 A CAMERA GRM2 GAIN i . O PHA3E ANGLE 35 30LAR ZENITH ANGLE 76 APPROX. WIDTHi 8 2 i KM HEIGHT 1294 KM RIZEAU
;~AR SAR DROPOUT GEOM 3TRETCH
12-11-69 142933 JPL/IPL FIG. 9. Enlargement of a portion of Mariner 7 Near Encounter picture No. 19.
7Mi9 DAY 2i? 04.53.04 A CAMERA GRM2 GAIM1.0 PHASE AMGLE 35 SOLARZEMITH AMGLE 76 A~. WIDTH 1821 KM HEIGHT1294 KM RIZEAU 3AR SAR 3AR
DROPOUT
r,IDRIHG MOI3REM DIFFPIC GEOM STRETCH i2-ii-69 i42645 JPL/IPL FiG. 10. Same as Fig. 9 after partial coherent noise removal.
7N19 DAY 217 04.53.04 A CAMERA GRN2 GAIN 1.0 PHASE ANGLE 35 SOLARZENITH ANGLE 76 APPROX. WIDTH i 8 2 i KM HEIGHT 1294 KM RIZEAU SAR SAR SAR DROPOUT NORING NOISREM DIFFPIC GEOM DIFFPIC STRETCH IS-tI-E,9
143346 JPL/IPL
FiG. 11. Difference picture obtained by subtracting the data shown in Fig. 9 from that given in Fig. 10. Zero differences appear as a neutral gray shade.
6N13 7 / 3 1 / 6 9 0 5 . 1 3 . 3 5 A CAMERA RED GAIN 1 . 0 PHASE ANGLE 51 SOLAR ZENITH AHGLE 61 APPROX. WIDTH 1107 KN HEIGHT 1139 KM SAR RIZEAU SRR SAR DROPOUT QSAR GEDM OSAR STRETCH 0 2 -1 8 -7 0 113601 JPLzIPL
FIG. 12. An enlarged portion of Mariner 6 Near Encounter picture number 13.
6H4_3 7/31_/69 0 ~ . 1 3 . 3 5 A C ~ RE~ F_~M 1 . 0 ~ AFIGLE ~1 SOLAR ZEMITH ANGLE 61 API=RO~. ~ T H 1107 KM HE'IGHT t t 3 9 KM SAR RIZEAO SAR SAR DROPOUT Q~ MORIMG MDI~EM PIXH PZXH STRETCH
FIG. 13. Same as Fig. 12 after coherent noise removal using the direct two-dimensional bandpass technique on an IBM 360/75 computer (see text).
PHA~E ANGLE 51 SOLARZEHITH A ~ E 61 APPROX. WIDTH iiO? KM HEIGHT ii39 KM SAR RIZEAU SAR SAR ~ROPOUT OSAR HDRIHG HnISREM PIXH PIXH DESTREAK STRETCH
FIG. 14. Same as Fig. 13 after A T R gain fluctuation removal ("destreak').
PHASE ANGLE 51 $DLAR ZEHITH ANGLE 61 APPRD×. WIDTH 1107 KM HEIGHT 1139 KM SAR RIZEAU SAR SAR DROPOUT QSAR
MORING MOISREM
PI×H PIXH DESTREAK
DESPIKE ~ZBgT CH
FIG. 15. Same as Fig. 14 after single pixel spike noise removal ("despike").
GN23
7/31/6@ 05.~0.37 A CAMERA GRN1 GAIN PHASE ANGLE 8 0 SOLAR Z E N I T H ANGLE 8 2 APPROX. WIDTH 8 9 3 KN H E I G H T 7 1 0 KH FILTER 3 T R E T C H GEOH
02-17-70
1.7
23W930
JPL/IPL
FIG. 16. An enlarged portion of Mariner 6 Near Encounter picture number 23 after contrast enhancement and sine wave response filtering for enhancement of fine detail.
GN23
7/31/C~B 05.20.3T A CAMERA GRNi GRIN l.Y P H A 3 E ANC4_E 8 0 3OLRR EENITH ANGLE 82 ~hc~:~KXIX. W I D T H 8 S 3 K M HEIGHT TiO KM 3mR RIZE~ 3 & R 34~R N O R I N G N O I 3 R E M D I F F P I C N O R I N G N O I 3 R E M DIFFPIC NORING NOI3REM DIFFPIC NORING NOI3REM OIFFPIC F I L T E R 3 T R E T C H C-EOM 02-27-70
23220~
JPL/IPL
FIG. 17. Noise r e m o v e d version of M a r i n e r 6 N e a r E n c o u n t e r picture n u m b e r 23 processed t h e s a m e way as t h e picture s h o w n in Fig. 12.
Mariner 1969televisionimage processing
267
like. The character of these noises is best illustrated in the frequency domain. Figure 6 shows Mariner 6 N.E, picture no. 18 and a two-dimensional Fourier transform of that same picture. The central region in the transform (DC is at the origin) contains most of the scene modulation, or picture power. The sharply defined, star-like maxima occurring throughout two-dimensional frequency space represent coherent noises. Figure 7 is a similar two-dimensional Fourier transform of a portion of Mariner 7 Near Encounter picture no. 19. In this transform, a different quadrant convention is employed than in Fig. 6, and DC is at the corners. Again, the presence of periodic components superimposed on the smoothly varying image "continuum" is quite evident. These well defined (in the frequency domain) peaks represent an almost trivial problem from the viewpoint of pattern recognition, and their isolation and removal from picture data is in principle quite straightforward. The quantitative and accurate removal of these periodic components can, however, be difficult in practice, because of their complex phase relationships and amplitude variation throughout the picture format. Noise removal requires first the acquisition of two-dimensional frequency information as shown in Fig. 7. On the basis of measurements made using this data, the picture is bandpass filtered, i.e. only those periodic components which one desires to remove are passed into an output picture. This output picture then is essentially the noise pattern which one wishes to extract from the original picture. The next step involves fitting this uniform, idealized pattern to the picture data so that the quantitatively correct amplitude can be subtracted in each area of the picture. The fitting is done by means of least squares analysis for small (30 × 30) portions of the picture. A modulation coefficient matrix is calculated which contains a coefficient for each small area so that, when the noise is multiplied by that coefficient and the result subtracted from the raw picture, the variance in that area of the output picture is minimized. The raw picture, noise pattern and modulation coefficient matrix are then combined in the final noise removal step. The operation of this process is shown in Figs. 7-11. Figure 8 is the two-dimensional Fourier transform of picture 7N 19 after noise removal. Note the vacant areas which are occupied by sharp maxima in the left hand transform. Figure 9 shows raw and Fig. 10 noise removal versions respectively of the same small area of 7N 19. Figure 1 1 is a difference picture obtained by subtracting the noise removed from the raw versions shown in Figs. 9 and 10. It represents then, the actual noise pattern extracted from this portion of 7N 19. Note the image detail apparent in the noise pattern. The presence of this non-periodic modulation may at first seem to suggest the passage of some "picture power" into the noise picture, which would imply the removal of some image detail along with noise. Actually, the presence of these features in the noise pattern is merely illustrative of the action of the modulation coefficient matrix, which found virtually no noise modulation in these nearly saturated portions of the picture. The improvement illustrated in Figs. 9 and 10 resulted, as can be seen by comparing Figs. 7 and 8, from the removal of a small number of frequency components. This limitation is imposed by the nature of the bandpass technique employed, which operates on a picture one line at a time because of core limitations. This requires a two step filtering procedure, once horizontal, and once vertical. The areas removed in two-dimensional frequency space are therefore determined by the intersections of bands projected parallel to the axes. In order to avoid the loss of image information, the number of such bands must be kept to a minimum, and therefore, only a few rectangles can be removed in a single "pass". We are presently employing a direct two-dimensional bandpass technique using a larger machine (IBM 360/75). This method permits the removal of virtuallv all periodic components in a
268
JAMES A. DONNE
single pass, and produces an output picture with a significantly lower residual noise content than is evident in Fig. 10. Illustrative of the direct two-dimensional noise removal and subsequent processing steps is the set given in Figs. 12-15. These figures show a portion of Mariner 6 N.E. picture No. 13 as it undergoes processing for coherent noise removal (Fig. 12), the removal of streaking caused by small ATR gain fluctuations (Fig. 13) and single pixel "spike" noise removal (Fig. 14). The removal of coherent noises enhances the visibility of fine detail in the pictures. Furthermore, more improvement is made possible by the removal of these noises, in that edge enhancement by means of convolution filtering becomes practical, once the sharply defined coherent noises are removed. Such filtering is not practical in the presence of the fine structured noises characteristic to the Mariner '69 data. An example of contrast enhancement and sine wave response filtering applied to Mariner 6 N.E. picture No. 23 is given in Fig. 17. Figure 12 shows the effects of the same processing applied to the raw (i.e. not processed for noise removal) version of that same picture. CONCLUSIONS The Mariner 1969 television image processing experience to date has provided a clear demonstration of the value of digital image processing in the systematic removal of vidicon system signatures from television image data received from distant spacecraft. Experience gained in performing image processing on selected Ranger, Surveyor and Mariner IV data provided the basis for the Mariner '69 processing library, and it is anticipated that the work reported in this paper will similarly benefit the Mariner 1971 orbiters and succeeding missions. Acknowledgement~The work described herein represents the collective efforts of T. C. RINDFLEISCH. W. D. STROMBERG, D. M. BROWN, R. M. RuIz, J. M. SOHA, A. R. GILLESPIE, H. J. FRIEDEN, D. J. MATHEWS, L. L. BRUMLEY, B. H. HAMILTONand the writer. In particular, the technical contributions of T. C. RINDFLEISCH,who conceived and implemented the majority of the Mariner 1969 processing algorithms are gratefully acknowledged, as are the guidance and cooperation of the Mariner 1969 TV Principal Investigator, Professor ROBERTLEIGHTON of the California Institute of Technology. This paper presents the results of one phase in research carried out at the Jet Propulsion Laboratory, California Institute of Technology, under Contract No. NAS 7-100, sponsored by the National Aeronautics and Space Administration.
PergamonPress 1970. Vol. 2, pp. 261-268. Printed in Great Britain
Mariner 1969 Television Image Processing J A M E S A. D U N N E Jet Propulsion Laboratory, 4800 Oak Grove Drive, Pasadena, California 91103, U.S.A.
(Received 24 February 1970) Abstract--Mariners 6 and 7 sent back approximately two hundred photographs of Mars, taken from distances ranging between one million and three thousand km relative to the planet surface. These images contain true object space information distorted, to one extent or another, by the signature of the imaging system itself. In order to maximize the scientific yield of the television data, the natures of the distortions introduced by the imaging system must be quantitatively determined, and techniques developed whereby they can be accurately removed. Television systems typically exhibit photometric nonuniformities and nonlinearities, geometric distortions, both optical and electronic, image retention and spatial frequency response limitations. In addition, Mariner 1969 specific problems were introduced in the course of on-board processing applied in the interest of data compression and etficient bandwidth utilization. A cubing amplifier and an automatic gain control circuit were employed, with photometric "DC reference" supplied in the form of a separately encoded, under sampled raw signal. The effects of on-board processing can in principle be removed by means of combining these separate data streams. Digital image processing techniques have been developed at the Jet Propulsion Laboratory which enable us to characterize these various distortions using calibration data, and to remove them from images returned by the two Mariner spacecraft. This paper will provide brief descriptions of some of these techniques and illustrations of their application to Mariner 1969 image data.
INTRODUCTION THIS paper will describe briefly the nature of the Mariner 1969 TV system and imagery, and some of the processing procedures being applied to the returned data in the Image Processing Laboratory (IPL) of the Jet Propulsion Laboratory's Space Sciences Division, a facility consisting of a dedicated IBM Model 360/44 computer and special purpose peripheral equipment, image scanning and display hardware and a software system designed for image manipulation and processing. The IPL is staffed with personnel specializing in image analysis, image processing technique development, hardware development and operational support. The major phases of the IPL support to the Mariner 1969 television experiment included preflight video subsystem calibration support, near-real-time flight image analysis for mission operations, and detailed post mission television image processing in support of the television experiment team. Preflight calibration activities included participation in the design and preparation of special procedures and test targets, preliminary calibration data analysis for engineering evaluation, and the detailed reduction of calibration data for the quantitative restoration of flight imagery. Inherent in the preflight activities was the development of image processing techniques and software for the correction of Mariner '69 unique video subsystem characteristics. The initial post mission image processing phase is, from the viewpoint of science data analysis, preliminary, representing as it does a prerequisite to the quantitative analysis of the Mariner TV data. Because of the non-linear contrast emphasis electronics inherent in the television camera system, digital processing is required before even approximately 261
~.
,
3,o
,~
s~
-
t
i
+++
F
t
i 4
-
. . . .
~
'
',LINE
++++l BY I LINE
1 ----~
:
EVERY7TH PIXEL, A/D TO 8 lilTS, 6 LEAST SIGNIFICANT BITS DIGITAL RECORD, 16.2 KBPS
ON EARTH
PIXEL = PICTURE ELEMENT M IV 2(]0 PIXELS BY 200 LINES
++* * DIGITAL DATA FROM OTHER EXPERIMENTS PLACED ON DIGITAL RECORDER
....
EVERY 28TH PIXEL, ,6./1:), 8 BITS, 4 CENTER BITS, DIGITAL ENCODE, PLACE ON ANALOG RECORDERDURING RETRACE TIME
SPACECRAFF
DIGITAL VIDEO
EVERY 28TM
TELEVISION DATA FLOW
FIG. 1. Mariner 1969 TV data flow.
TYPICAL LINE OF VIDEO
RETRACE TIME FREQ. 18.9 KHz
( 60 MSEC/LINE) (704 LINES/PICTURE ) = 42.24 SEC/PICTURE
i
. . . . . .
945 PIXELS TELEVISION FRAME FORMAT
,
~EC
.......... 945 PIXELS/50MSEC, AGC, PREEMPHASIS, ANALOG RECORD, 18.9 KHz CARRIER
j,
-°
,,,,i
z
~o
............ ~............ i ........... J.......... ~........... ---',
i,o
ACTIVE LINE TIME
LINE READOUT TIME: 60 MSEC
DSS
DATA STORAGE SUBSYSTEM
ANALOG VIDEO
z=
Z
Mariner 1969televisionimageprocessing
263
quantitative intensity measurements can be made on the returned pictures. The full realization of the scientific objectives of the television experiment requires additionally careful correction for systematic noises, photometric non-uniformities and non-linearities, geometric distortions, and modulation transfer function degradation, all of which constitute the signature of the television subsystem. The series of procedures designed to remove this signature from the images returned by Mariners 6 and 7, referred to collectively as decalibration and cleanup, will be discussed herein. MARINER '69 IMAGE DATA FORMATS A simplified data flow diagram of the Mariner 1969 television system is given in Fig. i. Major subsystems are the TV subsystem (TVS), including the cameras, head electronics, video amplifiers, cuber, automatic gain control (AGC), automatic aperture control (AAC) and the data storage subsystem, including the analog tape recorder (ATR), analog channel A/D converter and digital tape recorder (DTR). Analog video is recorded in the analog mode on the ATR and digitized to six bits on playback, whereas digital video is digitized before recording on the DTR or transmission back to Earth in real time. Composite analog video (CAV) passes through an X 3 amplifier (cuber) for contrast enhancement, before being recorded on the ATR. Near encounter CAV, in addition to being acted upon by the cuber, is subject to two separate electronic gain adjustments. First, the automatic aperture control (AAC) sets one of eight gain states, depending on the signal received from the AAC sense area in a previous A camera frame. Second, an automatic gain control (AGC) amplifier, acting in real time, continuously adjusts gain to keep the CAV signal on the most sensitive portion of the cuber transfer curve. AAC gain state information is transmitted in the science data stream, but no information is available on the state of the AGC, and therefore, near encounter CAV contains no photometric DC reference. In the far encounter mode, the automatic aperture (AAC) control is locked in its lowest gain state, and the automatic gain control (AGC) is inhibited. Every seventh sample of the video data is digitized directly by a separate A/D converter in the television subsystem and encoded to eight bits, of which the six least significant bits (LSB) are either recorded on the digital tape recorder (DTR) and transmitted or transmitted directly in real time. In addition, every twenty-eighth (/28) sample is recorded on the ATR embedded in the CAV during the television retrace time. The/28 words contain only the four central bits, i.e. two LSB and two MSB are truncated. Every seventh digital video (DV) frames also contain science telemetry introduced during the active line scan producing a "data bar" corresponding to the central 26 samples of each DV line. Therefore, of 134 possible DV samples per line, only 108 are transmitted. Near encounter DV is affected by the AAC gain state but not by the AGC amplifier and can thus be used as a reference for photometric restoration. Mariner 6 returned 50 Far Encounter CAV pictures and 25 Near Encounter CAV pictures. Mariner 7 returned 93 and 35 far and near encounter CAV pictures, respectively. Of the 143 total F.E. CAV pictures, corresponding DV data is available for approximately 10. All of the N.E. pictures were, of course, received in the form of CAV and DV. In addition, some 3100 real time F.E. DV pictures were transmitted by the two spacecraft. CALIBRATION DATA The goal of the television system calibrations was the acquisition of data by means of which system performance parameters can be quantitatively characterized. Data of this
264
JAMESA. DUNNE
type is useful in two areas. First of all, they provide a basis for the selection of flight subsystems and the prediction of their response during the various phases of mission operations. Secondly, they are necessary for the development and application of image processing techniques for the quantitative restoration of image intensity and geometry information. The general categories of system performance for which calibration data must be collected include noises and photometric, geometric and modulation transfer function distortions. In each of these principal categories, data is required to characterize a number of independently varying parameters. Photometric distortions, for example, include spatial sensitivity functions for both primary and residual images, veiling glare introduced by light scattering in the optical system, shutter leaks and various electronic transfer functions and gain states. TVS calibration data was obtained on the bench, in the thermal vacuum chamber and on the spacecraft, the latter in the Spacecraft Assembly Facility (SAF) and at the Eastern Test Range (ETR). Calibration tests consisted of near and far encounter mode flat fields, geometric distortion and modulation transfer function targets, complex grey scale targets viewed in various orientations to provide residual image data, simulated lunar limb and low contrast surface scenes, orbit determination test targets, narrow bandpass filter targets for spectral response characterization and miscellaneous special tests designed to provide data on light leaks, out of field scattering, and the like. Nearly six thousand calibration frames were recorded in all, including proof test model and flight spare system calibration. SUMMARY OF DECALIBRATION PROCESSING PROCEDURES Because of the unique design of the Mariner 1969 television system, photometric decalibration must await the completion of the reconstruction process, which provides an output picture in which the effects of the on-board processing electronics have been removed. This process will be discussed in some detail in a succeeding section. Reconstructed photographs contain all of the photometric distortions characteristic to vidicon images. The first of these to be removed in the decalibration procedure is that of residual images. Calibration data provides a measure of retentivity as a function of previous exposure and position in the image format for each of the flight cameras. This information permits the construction of a residual image sensitivity map for each of the cameras, which can be used to modulate the previous picture, the resultant then being subtracted from the photograph containing the residual image to effect the desired correction. Far Encounter pictures contain residual images formed by the immediately previous exposure. However, only widely spaced F.E. pictures were recorded in order that the limited (33 pictures per tape load) capacity of the ATR could be efficiently utilized. The technique for residual image removal for Far Encounter, therefore, involves the use of subtracting the appropriately scaled portion of the picture from itself, after first translating it to coincide with the location of the residual image. The registration of the real to residual image can be accurately checked using difference picture techniques. Since the planet revolves only about ~° per frame time, and the residual image percentage is less than 12 per cent of the previous exposure throughout the image format, photometric errors introduced in this procedure are small. Photometric decalibration also involves the removal of coherent noises superimposed on the video signal, a procedure which will be described in a later section. After noise and residual image removal, correction must be made for sensitivity variations across the scanned area of the vidicon target. Calibration data provides a light
Mariner 1969televisionimageprocessing
265
transfer characteristic function (data number out vs. luminance in) for each point in the image format. These data are obtained using luminance sources specially constructed to be uniform to within a few tenths of one per cent over the entire region viewed by the vidicon target. Spatial variations in output intensity can therefore be accurately described in terms of sensitivity variations on the vidicon surface. Calibration fiat field pictures, properly scaled for temperature coefficients and spectral response, provide the basis for the final sensitometric correction of the flight data. The removal of geometric distortions from the flight data is required in order to allow the accurate determination of spatial relationships and scaling in object space. These types of measurements are important in attempts to construct maps of the areas photographed for the guidance of future exploration efforts, and for the determination of the figure of the planet in using Far Encounter photography. Two types of distortions exist in the data. There is optical distortion present in the image on the vidicon target, as in the case of any camera system. A more significant distortion, termed "electronic", is introduced in the course of scanning the target to produce the video signal. These effects can be separated and individually corrected, by means of in situ measurements of a reseau pattern which was accurately deposited on the vidicon target, combined with calibration data taken using a test target produced from grating engine-scribed glass plate. The procedure followed in Mariner '69 geometric correction involves first the normalization of the flight data, based on reseau locations determined for each individual picture, to the same target-space coordinates as a selected geometric distortion calibration frame. A second geometric correction then is conducted using parameters which have been previously determined to correct the image of geometric distortion test target to its independently measured true shape. Another correction which is made to the data involves the restoration of high spatial frequency information by means of convolution filtering. The filters, referred to as sine wave response filters in recognition of the generally sinusoidal intensity distribution of the scanning beam, are derived from calibration measurements of the modulation transfer function of the camera systems obtained using computer generated variable frequency low contrast sinusoidal targets. This brief summary of decalibration procedures touches only the major classes of processing steps, and is intended only to serve as a background for the ensuing discussion of the two procedures which dominate the Mariner 1969 decalibration processing at the Image Processing Laboratory ; reconstruction and coherent noise removal. RECONSTRUCTION As described in earlier sections, the fully sampled Near Encounter data requires the restoration ofa DC photometric reference before measurements of relative scene luminances can be made. The data formats obtained from the spacecraft are illustrated in Fig. 2. Taken counterclockwise beginning in the upper right hand corner, these are every twentyeighth (/28) digital video, every seventh (DV) digital video and composite analog video (CAV). The picture in the lower right hand corner of Fig. 2 represents the product obtained in the reconstruction process. The mottled appearance of the DV and/28 pictures results from the truncation of the two most significant bits (MSB) before transmission. The first and most difficult step in the reconstruction process involves the restoration of these bits. MSB restoration is an iterative operation, wherein automatic computations are interleaved with manual adjustments, until a satisfactory product is obtained. The primary tool in
266
JAMESA. DUNNE
accomplishing MSB restoration is a program which, in the course of several passes through the picture, attempts to restore MSB's, using criteria which essentially demand smooth intensity transitions in the output picture. Since there are two missing MSB's, the true intensity at any given point in the picture can be in any of four "quadrants" i.e. 00, 01, 10 or 11. The MSB program, then, first scans the columns in the picture looking for sharp discontinuities, which represent probable quadrant changes. It also looks for lesser discontinuities, termed "ambiguities" at which a quadrant change may or may not have occurred. The program then selects a column which exhibits the largest number of quadrant changes and the smallest number of ambiguities, and begins to correlate across rows in the picture, again using the smoothness criterion to make most probable MSB assignments. The output picture so produced is then examined by an analyst, errors in MSB assignments noted, and point-by-point adjustments made to correct MSB assignment errors. This procedure often takes several iterations, particularly in the case of pictures wherein noise bursts have produced anomalous intensity values which perturb the MSB program. Obviously, also, certain true image characteristics (i.e. small, sharp crater shadows) can violate smoothness criteria and therefore require manual adjustment of the automatic MSB assignments to produce a true representation of the scene. This, of course, implies the use of the CAV pictures as a guide in the generation of correctly restored DV. An example of MSB restoration as applied to Mariner 7 N.E. picture No. 20 is given in Fig. 3. Once MSB restoration is complete for both DV and/28, these pictures are combined with the CAV to produce the reconstructed picture. The technique used here is one of multiplicative adjustment of the CAV intensity to match that of the MSB restored digital value at the corresponding point in the picture. The gain factor used to make this adjustment is then applied to adjacent CAV points, after correction by linear interpolation to vary smoothly between digital video sampled points. The product of this process is, once again, iterated upon by a combination of hand adjustments and discontinuity-recognizing programs, to produce the final reconstructed picture. Iterations are required because of errors in either CAV or DV introduced by noise and/or phase shifts. Figures 4 and 5 illustrate the effects of the reconstruction process. They are mosaics of five Mariner 7 wide angle camera pictures of the Martian south polar cap, Fig. 4 being raw CAV, and Fig. 5 the reconstructed versions of these same photographs. Note the photometric distortions in Fig. 4, the luminance of the desert on the right appearing to be nearly equivalent to that of the "snow" mantled polar cap region. The true luminance relationships are shown in Fig. 5. Note also the dark collar around the cap-desert interface. This dark band is illustrative of the time constant of the AGC. The actual position of the sunset termthator shows up, on the other hand, more clearly at the left edge of the raw mosaic than in the reconstructed one. These pictures are also illustrative of the value of the on-board processing electronics in enhancing visible detail in the raw pictures. The true contrast range on Mars (Fig. 5) is obviously too great to allow the optimum visibility of surface detail over the whole of the region covered by these mosaics in a single photographic print. Finally, it should be noted that the residual images of earlier limb pictures visible in Fig. 4 can only be removed after the completion of the reconstruction procedure since this process relies on a knowledge of the true exposure level in the picture which produced the residual image. COHERENT NOISE REMOVAL Mariner '69 data is characterized by the presence of a significant amount of structured noise introduced into the video signal by pickup from power supplies, heater wires and the
I~[t~l.ff-iM l Y l ~ , t . ~ $ P t , / I I ~
,
FxG. 2. Data formats. Clockwise from lower left, composite analog video, /7 and /28 digital video and reconstruc~,ed picture.
[facing p. 266
FIG. 3. MSB restoration for /7 digital video frame.
FIG, 4. Raw CAV mosaic of the Mariner 7 polar cap sequence.
FIG. 5. Mosaic of reconstructed versions of the Mariner 7 wide angle camera polar cap sequence.
i
08-31-69 110410 JPL/IPL
09-04-69 085843 JPL/IPL
FIG. 6. Mariner 6 Near Encounter picture number 18 and its two-dimensional Fourier transform.
PICTURE COUMT 6, S/C 63, STATIOM 21, CAMERA ID DAY 213, GMT 02195139 STRETCH
PICTURE COUNT 6, S/C 63, STATIOM 21, CAMERA ID i DAY 2±3, GMT 02195139 STRETCH 518X512 FOURIER TRAMSFORM FFT2 FFTPIC LINEAR AMPLITUDE/AMPLITUDE
7M19 DAY 217 04.53.04 A CAMERA GRM2 GAIM 1.0 PHASE AMGLE 35 SDLAR ZEMITH AMGLE 76 APPROX. WIDTH iSZi KM HEIGHT 1294 KM RIZEAU SAR FFT2 FFTPIC LIMEAR CPIC AMPLITUDE 12-06-69 180614 JPL/IPL FIG. 7. Two-dimensional Fourier t r a n s ~ r m of Mariner 7 Near Encounter picture number 19.
7N19 DAY 217 04.53.04 A CAMERA GRN2 GAIN 1.0 PHA3E ANGLE 35 SOLAR ZENITH ANGLE 76 APPROX. WIDTH 1821 KM HEIGHT1294 KM RIZEAU SAR SAR ~AR DROPOUT NORING NOISREM DIFFPIC FFT2 FFTPIC LIMEAR CPIC AMPLITUDE 12-09-69 111739 JPL/IPL FIG. 8. Two-dimensional Fourier transform of Mariner 7 Near Encounter picture
number 19 after partial removal of coherent noises.
7M19 DAY 217 04.53.04 A CAMERA GRM2 GAIN i . O PHA3E ANGLE 35 30LAR ZENITH ANGLE 76 APPROX. WIDTHi 8 2 i KM HEIGHT 1294 KM RIZEAU
;~AR SAR DROPOUT GEOM 3TRETCH
12-11-69 142933 JPL/IPL FIG. 9. Enlargement of a portion of Mariner 7 Near Encounter picture No. 19.
7Mi9 DAY 2i? 04.53.04 A CAMERA GRM2 GAIM1.0 PHASE AMGLE 35 SOLARZEMITH AMGLE 76 A~. WIDTH 1821 KM HEIGHT1294 KM RIZEAU 3AR SAR 3AR
DROPOUT
r,IDRIHG MOI3REM DIFFPIC GEOM STRETCH i2-ii-69 i42645 JPL/IPL FiG. 10. Same as Fig. 9 after partial coherent noise removal.
7N19 DAY 217 04.53.04 A CAMERA GRN2 GAIN 1.0 PHASE ANGLE 35 SOLARZENITH ANGLE 76 APPROX. WIDTH i 8 2 i KM HEIGHT 1294 KM RIZEAU SAR SAR SAR DROPOUT NORING NOISREM DIFFPIC GEOM DIFFPIC STRETCH IS-tI-E,9
143346 JPL/IPL
FiG. 11. Difference picture obtained by subtracting the data shown in Fig. 9 from that given in Fig. 10. Zero differences appear as a neutral gray shade.
6N13 7 / 3 1 / 6 9 0 5 . 1 3 . 3 5 A CAMERA RED GAIN 1 . 0 PHASE ANGLE 51 SOLAR ZENITH AHGLE 61 APPROX. WIDTH 1107 KN HEIGHT 1139 KM SAR RIZEAU SRR SAR DROPOUT QSAR GEDM OSAR STRETCH 0 2 -1 8 -7 0 113601 JPLzIPL
FIG. 12. An enlarged portion of Mariner 6 Near Encounter picture number 13.
6H4_3 7/31_/69 0 ~ . 1 3 . 3 5 A C ~ RE~ F_~M 1 . 0 ~ AFIGLE ~1 SOLAR ZEMITH ANGLE 61 API=RO~. ~ T H 1107 KM HE'IGHT t t 3 9 KM SAR RIZEAO SAR SAR DROPOUT Q~ MORIMG MDI~EM PIXH PZXH STRETCH
FIG. 13. Same as Fig. 12 after coherent noise removal using the direct two-dimensional bandpass technique on an IBM 360/75 computer (see text).
PHA~E ANGLE 51 SOLARZEHITH A ~ E 61 APPROX. WIDTH iiO? KM HEIGHT ii39 KM SAR RIZEAU SAR SAR ~ROPOUT OSAR HDRIHG HnISREM PIXH PIXH DESTREAK STRETCH
FIG. 14. Same as Fig. 13 after A T R gain fluctuation removal ("destreak').
PHASE ANGLE 51 $DLAR ZEHITH ANGLE 61 APPRD×. WIDTH 1107 KM HEIGHT 1139 KM SAR RIZEAU SAR SAR DROPOUT QSAR
MORING MOISREM
PI×H PIXH DESTREAK
DESPIKE ~ZBgT CH
FIG. 15. Same as Fig. 14 after single pixel spike noise removal ("despike").
GN23
7/31/6@ 05.~0.37 A CAMERA GRN1 GAIN PHASE ANGLE 8 0 SOLAR Z E N I T H ANGLE 8 2 APPROX. WIDTH 8 9 3 KN H E I G H T 7 1 0 KH FILTER 3 T R E T C H GEOH
02-17-70
1.7
23W930
JPL/IPL
FIG. 16. An enlarged portion of Mariner 6 Near Encounter picture number 23 after contrast enhancement and sine wave response filtering for enhancement of fine detail.
GN23
7/31/C~B 05.20.3T A CAMERA GRNi GRIN l.Y P H A 3 E ANC4_E 8 0 3OLRR EENITH ANGLE 82 ~hc~:~KXIX. W I D T H 8 S 3 K M HEIGHT TiO KM 3mR RIZE~ 3 & R 34~R N O R I N G N O I 3 R E M D I F F P I C N O R I N G N O I 3 R E M DIFFPIC NORING NOI3REM DIFFPIC NORING NOI3REM OIFFPIC F I L T E R 3 T R E T C H C-EOM 02-27-70
23220~
JPL/IPL
FIG. 17. Noise r e m o v e d version of M a r i n e r 6 N e a r E n c o u n t e r picture n u m b e r 23 processed t h e s a m e way as t h e picture s h o w n in Fig. 12.
Mariner 1969televisionimage processing
267
like. The character of these noises is best illustrated in the frequency domain. Figure 6 shows Mariner 6 N.E, picture no. 18 and a two-dimensional Fourier transform of that same picture. The central region in the transform (DC is at the origin) contains most of the scene modulation, or picture power. The sharply defined, star-like maxima occurring throughout two-dimensional frequency space represent coherent noises. Figure 7 is a similar two-dimensional Fourier transform of a portion of Mariner 7 Near Encounter picture no. 19. In this transform, a different quadrant convention is employed than in Fig. 6, and DC is at the corners. Again, the presence of periodic components superimposed on the smoothly varying image "continuum" is quite evident. These well defined (in the frequency domain) peaks represent an almost trivial problem from the viewpoint of pattern recognition, and their isolation and removal from picture data is in principle quite straightforward. The quantitative and accurate removal of these periodic components can, however, be difficult in practice, because of their complex phase relationships and amplitude variation throughout the picture format. Noise removal requires first the acquisition of two-dimensional frequency information as shown in Fig. 7. On the basis of measurements made using this data, the picture is bandpass filtered, i.e. only those periodic components which one desires to remove are passed into an output picture. This output picture then is essentially the noise pattern which one wishes to extract from the original picture. The next step involves fitting this uniform, idealized pattern to the picture data so that the quantitatively correct amplitude can be subtracted in each area of the picture. The fitting is done by means of least squares analysis for small (30 × 30) portions of the picture. A modulation coefficient matrix is calculated which contains a coefficient for each small area so that, when the noise is multiplied by that coefficient and the result subtracted from the raw picture, the variance in that area of the output picture is minimized. The raw picture, noise pattern and modulation coefficient matrix are then combined in the final noise removal step. The operation of this process is shown in Figs. 7-11. Figure 8 is the two-dimensional Fourier transform of picture 7N 19 after noise removal. Note the vacant areas which are occupied by sharp maxima in the left hand transform. Figure 9 shows raw and Fig. 10 noise removal versions respectively of the same small area of 7N 19. Figure 1 1 is a difference picture obtained by subtracting the noise removed from the raw versions shown in Figs. 9 and 10. It represents then, the actual noise pattern extracted from this portion of 7N 19. Note the image detail apparent in the noise pattern. The presence of this non-periodic modulation may at first seem to suggest the passage of some "picture power" into the noise picture, which would imply the removal of some image detail along with noise. Actually, the presence of these features in the noise pattern is merely illustrative of the action of the modulation coefficient matrix, which found virtually no noise modulation in these nearly saturated portions of the picture. The improvement illustrated in Figs. 9 and 10 resulted, as can be seen by comparing Figs. 7 and 8, from the removal of a small number of frequency components. This limitation is imposed by the nature of the bandpass technique employed, which operates on a picture one line at a time because of core limitations. This requires a two step filtering procedure, once horizontal, and once vertical. The areas removed in two-dimensional frequency space are therefore determined by the intersections of bands projected parallel to the axes. In order to avoid the loss of image information, the number of such bands must be kept to a minimum, and therefore, only a few rectangles can be removed in a single "pass". We are presently employing a direct two-dimensional bandpass technique using a larger machine (IBM 360/75). This method permits the removal of virtuallv all periodic components in a
268
JAMES A. DONNE
single pass, and produces an output picture with a significantly lower residual noise content than is evident in Fig. 10. Illustrative of the direct two-dimensional noise removal and subsequent processing steps is the set given in Figs. 12-15. These figures show a portion of Mariner 6 N.E. picture No. 13 as it undergoes processing for coherent noise removal (Fig. 12), the removal of streaking caused by small ATR gain fluctuations (Fig. 13) and single pixel "spike" noise removal (Fig. 14). The removal of coherent noises enhances the visibility of fine detail in the pictures. Furthermore, more improvement is made possible by the removal of these noises, in that edge enhancement by means of convolution filtering becomes practical, once the sharply defined coherent noises are removed. Such filtering is not practical in the presence of the fine structured noises characteristic to the Mariner '69 data. An example of contrast enhancement and sine wave response filtering applied to Mariner 6 N.E. picture No. 23 is given in Fig. 17. Figure 12 shows the effects of the same processing applied to the raw (i.e. not processed for noise removal) version of that same picture. CONCLUSIONS The Mariner 1969 television image processing experience to date has provided a clear demonstration of the value of digital image processing in the systematic removal of vidicon system signatures from television image data received from distant spacecraft. Experience gained in performing image processing on selected Ranger, Surveyor and Mariner IV data provided the basis for the Mariner '69 processing library, and it is anticipated that the work reported in this paper will similarly benefit the Mariner 1971 orbiters and succeeding missions. Acknowledgement~The work described herein represents the collective efforts of T. C. RINDFLEISCH. W. D. STROMBERG, D. M. BROWN, R. M. RuIz, J. M. SOHA, A. R. GILLESPIE, H. J. FRIEDEN, D. J. MATHEWS, L. L. BRUMLEY, B. H. HAMILTONand the writer. In particular, the technical contributions of T. C. RINDFLEISCH,who conceived and implemented the majority of the Mariner 1969 processing algorithms are gratefully acknowledged, as are the guidance and cooperation of the Mariner 1969 TV Principal Investigator, Professor ROBERTLEIGHTON of the California Institute of Technology. This paper presents the results of one phase in research carried out at the Jet Propulsion Laboratory, California Institute of Technology, under Contract No. NAS 7-100, sponsored by the National Aeronautics and Space Administration.