|Imagery Examples||Before and after examples|
|Quickstart||Get started with a Python-based quickstart tutorial|
|Task Runtime||Benchmark runtimes for the algorithm|
|Input Options||Required and optional task inputs|
|Outputs||Task outputs and example contents|
|Advanced Options||Additional information for advanced users|
|Known Issues||Issues users should be aware of|
Advanced Image Preprocessor can be run with Python using gbdxtools .
This script uses AOP_Strip_Processor to produce orthorectified and atmospherically compensated multispectral and panchromatic images from a raw WorldView-3 (WV03) image over Naples, Italy.
# Quickstart Example producing an Orthorectified and AComp output for MS + PAN # First Initialize the Environment from gbdxtools import Interface gbdx = Interface() # Make sure both pan sharpening and DRA are disabled in order to get separate PAN and MS outputs. # The data input and output lines must be edited to point to an authorized customer S3 location) data = 's3://gbd-customer-data/CustomerAccount#/PathToImage/' aoptask = gbdx.Task('AOP_Strip_Processor', data=data, enable_pansharpen=False, enable_dra=False) workflow = gbdx.Workflow([ aoptask ]) #Edit the following line(s) to reflect specific folder(s) for the output file (example location provided) workflow.savedata(aoptask.outputs.data, location='S3 gbd-customer-data location/<customer account>/output directory') workflow.execute() print workflow.id print workflow.status
data variable specifies the S3 URL of the raw image.
gbdx.Task() call creates the
aoptask instance and specifies its desired input values.
workflow.savedata() call specifies that the task output will be saved in
Running the script in an IPython session produces these results:
In : # Quickstart Example producing an Orthorectified and AComp output for MS + PAN In : # First Initialize the Environment In : from gbdxtools import Interface In : gbdx = Interface() 2016-06-24 16:29:53,856 - gbdxtools - INFO - Logger initialized In : # Make sure both pan sharpening and DRA are disabled in order to get separate PAN and MS outputs. In : data = "S3 gbd-customer-data location/<customer account>/input directory" In : aoptask = gbdx.Task('AOP_Strip_Processor', data=data, enable_pansharpen=False, enable_dra=False) In : workflow = gbdx.Workflow([ aoptask ]) In : workflow.savedata(aoptask.outputs.data, location='S3 gbd-customer-data location/<customer account>/output directory') In : workflow.execute() Out: u'4362772047134837472' In : print workflow.id 4362772047134837472
These are the average runtimes for this algorithm. All benchmark tests were run using a standard set of images, based on our most common customer scenarios. Runtime benchmarks apply to the specific algorithm, and don’t represent the runtime of a complete workflow.
|Sensor Name||Total Pixels||Total Area (k2)||Time(secs||Time/Area k2|
The following table lists the Advanced Image preprocessor inputs.
All inputs are optional with default values, with the exception of
data which specifies the task's input data location. Click on the name for a more detailed description of the use case.
|data||N/A||S3 URL||S3 location of 1B input data.|
|enable_acomp||true||true, false||Run atmospheric compensation.|
|enable_pansharpen||true||true, false||Pan sharpen multispectral data.|
|enable_dra||true||true, false||Apply dynamic range adjustment.|
|enable_tiling||false||true, false||Tile output images according to the |
|bands||Auto||PAN+MS, PAN, MS, Auto||Bands to process. |
|parts||All Parts||Comma-separated part numbers||List of strip parts to include in processing.|
|ortho_epsg||EPSG:4326||EPSG codes, UTM||EPSG code of projection for orthorectification. |
|ortho_dem_specifier||SRTM90||S3 URL path to custom DEM||Runs in default mode unless a custom DEM is specified.|
|ortho_pixel_size||Auto||Pixel size in meters, Auto||Pixel size of orthorectified output. |
|ortho_tiling_scheme||N/A||Ex: DGHalfMeter:18||Tiling scheme and zoom level for orthorectification. Overrides |
|ortho_interpolation_type||Cubic||Nearest, Bilinear, Cubic||Pixel interpolation type for orthorectification.|
|dra_mode||IntensityAdjust||IntensityAdjust, BaseLayerMatch||Dynamic range adjustment type. |
|dra_low_cutoff||0.5||0.0 - 100.0||Low cutoff percentage for |
|dra_high_cutoff||99.95||0.0 - 100.0||High cutoff percentage for |
|dra_gamma||1.25||Nonzero float value||Gamma value for |
|dra_bit_depth||8||8, 16||Output bit depth for |
|dra_baselayer_prefix||s3://dg-baselayer/v2/||S3 URL||S3 location of base layer for |
|tiling_zoom_level||12||Integer (see tiling scheme)||Zoom level (i.e. size) of output tiles for |
The Advance Image Preprocessor task inputs can be set in various combinations to generate several different types of output imagery:
- Pansharpened and DRA 8-bit RGB image with atmospheric compensation
- Default behavior, no need to change inputs
aoptask = gbdx.Task('AOP_Strip_Processor', data=data)
- Multispectral image only (4-band or 8-band, depending on sensor) with AComp
aoptask = gbdx.Task('AOP_Strip_Processor', data=data, bands='MS', enable_pansharpen=False, enable_dra=False)
- Multispectral and panchromatic images (separate) with AComp
aoptask = gbdx.Task('AOP_Strip_Processor', data=data, enable_pansharpen=False, enable_dra=False)
- Panchromatic image only (AComp not available)
aoptask = gbdx.Task('AOP_Strip_Processor', data=data, bands='PAN', enable_acomp=False, enable_pansharpen=False, enable_dra=False)
Please pay attention to the following important notes.
- To get separate PAN and MS outputs:
datamust include both PAN and MS bands.
falseto skip combining of PAN and MS data.
falsebecause it relies on pansharpening to run first.
bandsmust be set to either
enable_tilingcan only be
ortho_epsgcan be set to
UTMwhich automatically determines the correct EPSG code for the UTM zone at the center of the input data.
- When set,
ortho_dem_specifierThe Default DEM (SRTM90) currently doesn't include any DEMs that cover the globe north of +60 degrees latitude. Additional non-standard DEMs may be available upon request. See Advanced Options to add your own DEM.
dra_modeis set to
BaseLayerMatch, a geographic projection must be used. Either
ortho_epsgmust be set to
ortho_tiling_schememust be set to use DGHalfMeter at some zoom level.
- For WorldView-1,
enable_dramust all be set to
falsesince that sensor only produces panchromatic data.
The Advanced Image Preprocessor outputs are the following.
|data||Y||S3 location where output is stored|
|log||N||S3 location where logs are stored|
The following table delineates some of the properties of the processing result output image in a variety of common task execution scenarios with multispectral data input. When multiple processing options are executed the order of operations is always Orthorectification -> AComp -> Pan-Sharpen -> DRA. Please note that not all processing combinations are captured in this table, for more information on advanced execution scenarios please refer to the Advanced Options section of this document.
|Name||Coordinate System||Band Grouping||Data Type||Example Use Case|
|ortho_epsg=EPSG:4326 enable_acomp=true enable_pansharpen=true enable_dra=true dra_mode=BaseLayerMatch||Geographic WGS84||3-Band (R,G,B)||8-Bit Unsigned Byte||Output may be used in larger mosaic as highest resolution RGB image available|
|ortho_epsg=EPSG:4326 enable_acomp=false enable_pansharpen=true enable_dra=false||Geographic WGS84||4-Band (B,G,R,N)||16-Bit Unsigned Integer||This 4-band output may be used in detection algorithms, however spectral integrity is compromised while resampling data from native resolution to panchromatic resolution. Further, 8-band pansharpening is not currently available and only the 'MS1' bands will be processed by this task|
|ortho_epsg=UTM enable_acomp=false enable_pansharpen=true enable_dra=false||UTM WGS84||4-Band (B,G,R,N)||16-Bit Unsigned Integer||This task is identical to the previous example, save for the additional projection of the data into Universal Transverse Mercator (UTM )with the WGS84 datum and corresponding UTM zone|
|ortho_epsg=EPSG:4326 enable_acomp=true enable_pansharpen=false enable_dra=false||Geographic WGS84||Output will match input e.g. "PAN+MS"||8-Bit Unsigned Byte||A multispectral image processed through Orthorectification and Atmospheric Compensation will be suitable for a landcover analysis|
|ortho_epsg=EPSG:4326 enable_acomp=false||Geographic WGS84||Output will match input e.g. "PAN+MS"||8-Bit Unsigned Byte||A multispectral image processed through Orthorectification and Atmospheric Compensation will be spatially accurate, but may not be suitable for spectral analysis without Atmospheric Compensation|
data output port contains the location where the AOP_Strip_Processor output is stored. Contents may vary slightly depending on the input settings. The following listing
is from the AWS CLI, with columns for the creation date, time, file size, and file name.
PRE in the size column indicates an S3 prefix, similar to a subdirectory. This is a typical result from running AOP_Strip_Processor with default settings. The sequence of characters "055421455010_01" is called a SOLI (Sales Order Line Item) and is an artifact of the image ordering system.
PRE 055421455010_01/ 2016-06-22 15:41:53 16 processed_strips.txt 2016-06-22 15:41:53 4200 workorder_055421455010_01.xml
055421455010_01/: Subdirectory containing the processing results.
processed_strips.txt: File containing a list of the images that were processed.
workorder_055421455010_01.xml: File containing the inputs that were used by the AOP workflow system.
This is a listing of the contents of the
PRE GIS_FILES/ 2016-06-22 15:41:26 4948 055421455010_01_assembly.IMD 2016-06-22 15:41:26 895166 055421455010_01_assembly.XML 2016-06-22 15:41:26 984030964 055421455010_01_assembly.tif
GIS_FILES/: Subdirectory copied from input data that contains various shapefiles describing the input image.
055421455010_01_assembly.IMD: IMD file (image metadata) copied from one part (scene) of the input data. Currently, no attempt is made to update this file to reflect the processed output.
055421455010_01_assembly.XML: XML file (additional metadata) copied form one part (scene) of the input data. Currently, no attempt is made to update this file to reflect the processed output.
055421455010_01_assembly.tif: The output image in GeoTIFF format.
You can also preview the results of AOP_Strip_Processor using the s3 browser.
log output port contains the location where a trace of log messages generated during processing is stored. This can be useful for debugging. A typical run would produce a listing similar to the one below.
2016-06-22 15:40:54 0 055421455010_01.stderr 2016-06-22 15:40:54 117469 055421455010_01.stdout
055421455010_01.stderr: Captured stderr stream from processing.
055421455010_01.stdout: Captured stdout stream from processing.
- The 'enable_acomp' option runs the DG Atmospheric Compensation Process. The default setting is on (True). This will remove haze and provide the best surface reflectance output for spectral analysis of imagery.
- The 'enable_pansharpen' output is a high-resolution RGB image. The process merges the lower resolution multispectral image with the higher resolution panchromatic image to produce a high resolution multispectral image (RGB). The default is to run pansharpening. It must be set to 'False' if you want preserve the full 8-band or 4-band image from the input image.
- Using the Custom DEM Option for 'ortho_dem_specifier'. The custom DEM must first be uploaded to the customer's S3 bucket; and you must specify the full path to the Custom DEM. Custom DEM data must be pre-processed to fit the DG Tiling Scheme using gdal_tiler, which automatically reprojects the DEM to EPSG:4326. No option needs to be executed to run in default mode (SRTM90). The table below describes the gdal_tiler inputs:
|data||Yes||any format recognized by gdal||S3 URL path to the input custom DEM including file name|
|tiling_scheme||Yes||"DGTiling"||tiles the DEM to be consistent with tiling used by the Image Preprocessor|
|zoom_level||Yes||Size Appropriate for DEM Resolution||"7" works for most cases|
|pixel_size||Yes||pixel size of DEM in degrees||Must be in degrees and set according to the custom DEM|
|compress||No||"on", "off"||highly recommended; saves storage space using a lossless compression algorithm|
Below is a QuickStart Script for gdal_tiler:
# Using gdal_tiler, creates tiles for a custom DEM as input for AOP from gbdxtools import Interface gbdx = Interface() # Edit the following path to reflect a specific path to the custom DEM data = "s3://gbd-customer-data/acct#/example.tif" # Run gdal_tiler prep task. The pixel_size must be set to the resolution of the DEM in degrees. Change as necessary gdaltiler = gbdx.Task("gdal_tiler", data=data, tiling_scheme="DGTiling", zoom_level="7", pixel_size="1", compress="on") workflow = gbdx.Workflow([ gdaltiler ]) # workflow.savedata(gdaltiler.outputs.data.value,location='customer directory used for the ortho_dem_specifier') workflow.execute() print workflow.id print workflow.status
- The default for 'enable_dra' is on (True) and it must be set to 'False' to produce a 4-band or 8-band image (+/- panchromatic band). If Pansharpening has been set to False, then DRA must also be manually set to False. For all other Dynamic Range Adjustment Settings: see below
- The 'enable_tiling' setting allows the image to be rendered according to a specified grid size. Tiling is used to improve performance of subsequent image processing steps in the workflow, especially when computing resources are limited. The default setting is off.
- 'bands' allows you to select the bands to be processes for further applications. The default is 'Auto', which will process all of the bands (including panchromatic) that are in the S3 input data location. Other options are PAN+MS, PAN, MS. Use when the next application of algorithm in your workflow requires specific band inputs.
partsinput can be used to limit processing to a subset of an input strip. This requires advance knowledge of the layout of a strip order. One way to get this information is by looking in the input strip's
GIS_FILESdirectory at the PRODUCT_SHAPE.shp vectors. That particular file shows the boundaries of each part (scene) of a strip. Once those numeric values are known, set
partsto a comma-separated list, e.g.
2, 3, 4.
- The 'ortho_epsg' The default is EPSG:4326 which is WGS84 geographic coordinates. For some cases, such as for change detection, square pixel are required so you must reproject the image to a UTM grid. You can specify the EPSG code if you know it, or set ortho_epsg='UTM' and the AOP processor will select the appropriate UTM zone.
- The output image pixel size can be specified in meters. The default setting is the same as the input pixel size ('Auto').
- A custom tiling scheme can be specified that overrides 'ortho_epsg' and 'ortho_pixel_size'
- This sets the resampling method applied during the AOP process. The default setting is ortho_interpolation_type='Cubic'. Other options are Bilinear and Nearest Neighbor. However, for spectral analysis Bilinear is preferred because it affects the spectral DN the least.
- The output zoom-level for viewing can be set when enable_tiling ='True'. The default is 12m.
The included DRA algorithm has several inputs that affect the final 8-bit RGB result. Please read all of the options carefully before making any adjustments. The default DRA setting will work best for most cases.
dra_mode- IntensityAdjust is for standalone, individual images that are not going to be mosaicked together. BaseLayerMatch uses a global base layer for color matching and helps maintain consistency when mosaicking. The base layer started out as color-balanced and mosaicked Landsat imagery but may have started incorporating higher-res imagery also.
dra_low_cutoff- Sets the black point in the histogram. Adjusting this will change the point in the histogram that is considered “black” and will darken and lighten the low end of the histogram. Setting the number lower (<0.5) will brighten the darker color tones and make the overall image lighter. Setting the number higher will saturate more of the darker color tones and make the overall image darker. Could start to lose detail in the darker areas if too heavy-handed. A light touch can make a huge difference so be careful.
dra_high_cutoff- Sets the white point in the histogram. Same general idea as the low cutoff percentage but operates on the brightest color tones to adjust what is considered saturated “white”. Setting this number lower (<99.95) will brighten the image while setting the number higher will darken the image. Again, a light touch is mandatory to keep things looking “normal”.
dra_gamma- Adjusts the curvature of the transfer function from input to output. When gamma=1, that is a straightforward, linear transfer from input to output. When gamma>1, the image will get overall brighter. Conversely, the image will get overall darker when gamma<1. Works in conjunction with the histogram cutoff values but is a completely independent parameter. Operates like a root stretch but with much finer adjustment settings. All three parameters, low cutoff, high cutoff, and gamma, work together to adjust the overall brightness, contrast, and dynamic range of the image. They’re all independent and will affect the final DRAed image in similar, but different, ways. Setting these is more an art than a science and it’s highly recommended to NOT mess with these unless the image is one of those special cases and is totally screwed up. Then the art comes into play.
dra_bit_depth- Typically it only makes sense to apply dynamic range adjustment to convert imagery to 8-bit. The 16-bit option is available mainly for debugging purposes, but isn't useful in normal situations.
- Custom DEM input does not work for polar stereographic projections.
- Custom DEM input is assumed to be in meters
If you have any questions or issues with this task, please contact [email protected] .
Updated over 1 year ago