Tuesday, June 22, 2010

Access your Oracle Database from Python

cx_Oracle is a free, downloadable Python extension module that enables Python to access an Oracle database. This can come in handy when you need to streamline or automate tasks that require the use of SQL statements to access components of your spatial database, such as feature classes and domains. The module is available for all combinations of a variety of Operating Systems, Oracle versions (9.2.0, 10.2.0, and 11.1.0), and Python versions (2.0 and up). *Note: An Oracle client (or server) installation is required in order to use cx_Oracle. If you do not require the tools that come with a full client installation, it is recommended to install the Instant Client which is far easier to install.

I found this module to be useful when deleting feature classes with domains from the database. In ArcGIS 9.3.1 and earlier versions, unused domains are left lingering in the database after feature classes have been deleted. What’s more, default domains have to be unregistered before they can be deleted. I wrote a script in Python using the cx_Oracle module to loop through all feature classes in an ArcSDE database, remove default domains from fields, delete each feature class, and then delete the domains remaining in the database. I’ve added this script to the Geospatial Database. If you would like to see an example of how to use cx_Oracle or are looking to delete feature classes with domains from a database like I described above, you can download the script here.

The script takes four arguments:
1) Owner: Data owner.
2) Password: password@NET SERVICE NAME
3) Workspace: The folder location of the sde database (and feature dataset, if applicable) containing feature classes to run the script on (ex. C:\Documents and Settings\user\Application Data\ESRI\ArcCatalog\example.sde)
4) SDE Owner: Owner of the SDE repository in the database (ex. sde)

Friday, April 30, 2010

What Is A Geospatial Analyst?

I came across this article that gives a good description of the roles of a Geospatial Analyst. I know that a lot of the traffic on my blog stems from people searching for a description of what a Geospatial Analyst does, so I thought I would share this. It mentions the skill sets required for a Geospatial Analyst position are environmental sciences, computer science, geography, and database design. I think this is a pretty inclusive list at a very high level (other than remote sensing, which is brushed upon later on in the article). Check it out here:

The Role of a Geospatial Analyst

Tuesday, March 23, 2010

"ezree"

Here’s an interesting tidbit: ESRI employees received an email from corporate last week regarding the pronunciation of ESRI. We will now be calling ourselves “ezree” instead of E-S-R-I to ensure “consistent name recognition around the world”. This makes sense as most non-ESRI employees know the industry’s leading GIS software company as “ezree”. Even I called it "ezree" prior to becoming an employee. Though now the pronunciation sort of makes me cringe. How do you feel about it?

Monday, February 22, 2010

Inserting Characters into a String - The Latest Addition to Batch Edit String Fields Toolbox

This post serves as a response to a comment posted by Josh on the Find and Replace entry. Josh commented:

"Brooke,
Let me ask a variant of Ivar's question. Ultimately I can resolve my situation the same way he identified above, however, is there a way to insert specified characters at a specific position. For instance; insert "ap" at position 2 from left for the field "Hpy" to create "Happy"? Any Ideas? Thanks. Nice site. Josh
"

Josh,
Thank you so much for your question and for visiting my blog. There are a couple ways of doing this:

1) You can use the field calculator in the layers attribute table using the following expression:
Left( [FieldName], 1) & "ap" & Right( [FieldName], Len( [FieldName]) -1)
So in this example, "ap" is added to the first character of the string (the Left function with the number 1 returns the first character), and then the characters of the original string minus the first character (the Right function with the number -1 returns all characters from the right of the original string minus the first character) are added to the end. So if the string in your field is "Hpy", "ap" will be added after the first character ("H"), and then the rest of the original string ("py") is added to the end. See the post on String functions for ArcMap Field Calculator for more VB examples for altering strings.

2) Thanks to your question, I added a new script to the Batch Edit String Fields toolbox I blogged about last September. Below is a graphic of this new tool.



The toolbox and scripts can be downloaded from the Geospatial Database. Use the link below to download the zip file. Ensure that the files are unzipped to the same folder (the scripts use relative paths).

Batch Edit String Fields.zip

Sunday, February 21, 2010

Using Python to Generate Status Emails for Lengthy Geoprocessing Tasks

In honor of PyCon (Python conference in downtown Atlanta) this weekend, I've decided to dedicate this post to a very valuable Python capability that can be extremely useful when running daily, behind-the-scenes scripts or scripts with lengthy processing times.

If there is a script that you run daily for some process such as an overnight synchronization, or an overnight cache, it can be a hassle checking the script's log file every morning to make sure it ran successfully. Or if there is a script you are running on an extremely large dataset with hundreds of thousands, or even millions of features, it would be nice to receive an email when the script completed that contained a subject providing information on whether the script completed successfully or stopped running due to some error, and in the message body see the error that caused the script to fail.

This can be done very easily with Python. I have uploaded a script to the Geospatial Database called PyEmail.py and it can be downloaded by right clicking on the link below and selecting "Save Target as...".

PyEmail.py

The uploaded script is written to work so that an email is generated once the script finishes running. If the script runs successfully, the email's subject displays "Script Completed Successfully!" and the message body is left blank. However, if the script fails, the subject displays "Script Failed", a log file with error messages is attached to the email, and the message body of the email notifies the user to check the attached log files for errors. The script can be easily modified to meet your own personal needs - maybe you only want to receive an email if the script fails, maybe you want the error messages to just print to the message body instead of having a log file, etc. All of these things are pretty simple changes within the script.

What the script requires from you:
1) Your own code which you will paste into the PyEmail script.
2) The path to your log file (if you are using a log file to catch error messages).
3) The email address(es) of the recipient(s).
4) The name and email address of the sender.
5) IP or name of the mail server used to send the email.
6) SMTP user name and password (if you are using SMTP authentification).

Wednesday, February 10, 2010

ET GeoWizards – A suite of tools extending ArcMap functionality

ET GeoWizards is a suite of GIS tools that extends ArcMap functionality and is available at all license levels. Additionally, the tools allow ArcView license holders to perform analysis tasks that would otherwise require an ArcEditor or ArcInfo license, or require the 3D Analyst extension. The full ET GeoWizards tool set without restrictions or limitations can be purchased here, however much of the functionality is absolutely free. Click here to download the tools for free (with some restrictions and data processing limitations without a license key) from ET SpatialTechniques. Installation includes a DLL that can be added to an ArcMap toolbar and provides an easy-to-use GUI with tabs of tools grouped by theme.

The installation package also includes an ET GeoWizards toolbox (.tbx) that contains tools that can be used in Model Builder and Command Line. For a complete list of free tools on both the GUI and in the toolbox, see the Free functions of ET GeoWizards page.

Below are *just a few* functions with descriptions from each tabbed group on the GUI that I personally have found useful. Definitely check out the complete list to see all of the free tools available.

Point
• Clean Point Layer – Deletes duplicate points.
• Measure Points – Assigns measures (M) or elevations (Z) to points based on interpolated values from the closest polyline within a user specified search tolerance.
• Reverse Geocoding – Uses a reference polyline layer to assign assign addresses to points of a point layer.



Polyline
• Clean Polyline Layer – Ensures topological correctness of polyline features.
• Get Z Characteristics – Calculates several Z dimension characteristics and stores them in fields added to the attribute table. Includes characteristics such as 2D and 3D length of the polyline, min and max Z values, and average and max slow uphill and downhill among others.
• Flip Polylines – Assigns the node closest to a specified start point as the from node so that the direction of all polyline features point away from the start point.



Polygon
• Clean Polygons – Ensures topological correctness of polygon features.
• Create Centerlines – Creates centerlines from polygon features. Can be created inside polygons (from polygon rivers and streets) our outside polygons (from parcel data).
• Get Adjacent Polygons – Stores the adjacent polygon names/IDs (user specified) as a comma delimited string as well as the count (number of adjacent polygons) for each polygon in the attribute table.



Convert
• Polyline to Point – Converts a polyline dataset to a point dataset at 1) vertices, 2) nodes, or 3) mid points.
• Point to Multipoint – Converts a point layer to a multipoint dataset.
• Point to Point Z (M) – Uses a numeric field to assign values to the Z or M dimensions.
• Polygon Z (M) to Point – Converts polygon vertices to points and adds attributes to store Z and/or M values of each vertex.
• Polyline Z (M) to Point – Converts polyline vertices to points and adds attributes to store Z and/or M values of each vertex.



Surface
• Build TIN – Builds a Triangulated Irregular Network from a point, polyline, or polygon feature layer.
• Features to 3D – Creates a 3D dataset from point, polyline, or polygon features by deriving the values from a surface (Raster or TIN) layer.



Geoproc
• Transfer Attributes – Transfers attributes from one polygon layer (source) to another (target) based on their spatial location.
• Spider Diagram (Spatial Location) – Creates a Spider Diagram representing the shortest distance between the points of a Center Points layer and the features in the destination layer (points, polylines, or polygons).
• Spider Diagram (Attribute Link) – Creates a Spider Diagram between the points of a Center Points layer and the features in the destination layer (points, polylines, or polygons) based on a common value in a user specified field in both layers.



Basic
• Redefine Fields – Change field names, length, precision, and scale.
• Order Fields – Exports a feature layer to a new feature class. The user selects the fields to be exported and the order in which they will appear in the attribute table.
• Vector Grid – Creates a polygon or polyline grid with user defined extents and cell size.



LinRef
• Create routes from existing polylines – Creates routes from existing polyline features. Input polylines that share a common identifier are merged to create a single route.
• Calibrate routes with points – Calibrates existing routes using the measures of existing points. The measures collected from the points can be M dimension values or can be stored in a ield of the point attribute table.
• Locate point features along routes – Finds the route and measure information of point features and creates a point event table. Search tolerance is used to correctly identify the closest routes to each point.
• Locate polygon features along routes – Calculates route and measure information at the geometric intersection of polygon data and route data and creates a line event table.



Misc
• Features To Envelopes – Creates a polygon from the envelope of each feature in the input feature class. Attributes of the original features are transferred to the envelope polygons.
• Features To Minimum Bounding Circles – Creates a circular bounding polygon from each feature in the input feature class. Attributes of the original features are transferred to the resulting polygons.
• Create Station Lines – Creates station lines along the polyline features of the input layer. The station lines are perpendicular to the original polylines with user specified station distance and length. The length can be constant or calculated from Z or M values (if the input layer has Z or M values).



In/Out
• Export To Google Earth – Converts all visible layers in a data frame to a Google Earth KML file.
• Import From Google Earth – Converts Google Earth (KML or KMZ) data in to feature classes.

Thursday, December 24, 2009

Reverse Geocoding with Google Maps

Here's a handy link for reverse geocoding. The service is provided by Digital Inspiration and uses Google Maps to return the address at the location of a single left mouse click. One thing to keep in mind: because the reverse geocoder uses the Google Maps API, the service is only as accurate as the street data in Google Maps.

Thursday, December 10, 2009

String Functions for ArcMap Field Calculator

Back in August I posted an entry on a suite of Python scripts that I wrote for altering strings in the ArcMap attribute table. This included scripts for capitalizing words, converting lower case to upper case and vice versa, trimming white spaces, and stripping characters from the beginning and end of strings. Click here to read that entry titled Batch Edit String Fields – Free Tools for Calculating String Fields and to download the toolbox and the corresponding scripts.

While the Python scripts have been very popular among followers of Geospatial Analyst, I’ve noticed that a lot of the traffic on my blog originates from people searching for the correct VB syntax to alter string attributes. Many people are just looking for the syntax to quickly alter a string field, instead of downloading and using the Python scripts that I’ve made available. So I’ve decided to write this entry on VB field calculator expressions for common string manipulations.

***The first thing to note: Many people mistakenly search for the syntax for LTrim() and RTrim() functions when they want to strip characters from the beginning or end of a string. However, the LTrim() and RTrim() functions simply remove preceding and trailing white spaces, respectively, from the beginning and end of the string. To strip characters (instead of white spaces) from a string, you must use the Right(), Left(), or Mid() functions.***

Listed below are expressions to plug into the field calculator along with the resulting output for the string “Hello Fantastic World!”.



Input: Hello Fantastic World!
Expression: Left( [FieldName], 5)
Output: Hello
Explanation: Returns the first 5 characters from the left.


Input: Hello Fantastic World!
Expression: Left( [FieldName], Len( [FieldName] ) -17)
Output: Hello
Explanation: Returns the string minus the last 17 characters.


Input:
Hello Fantastic World!
Expression: Right( [FieldName], 16)
Output: Fantastic World!
Explanation: Returns the last 16 characters from the right.


Input: Hello Fantastic World!
Expression: Right( [FieldName], Len( [FieldName]) -6)
Output: Fantastic World!
Explanation: Returns the string minus the first 6 characters.


Input: Hello Fantastic World!
Expression: Replace( [FieldName], "Fantastic", "Beautiful")
Output: Hello Beautiful World!
Explanation: Replaces the first string with the second string.


Input:
Hello Fantastic World!
Expression: Mid( [FieldName], 7, 9)
Output: Fantastic
Explanation: Returns the first nine characters beginning at the 7th character.


Input:
Hello Fantastic World!
Expression: Mid( [FieldName], 17, 5)
Output: World
Explanation: Returns the first 5 characters starting at the 17th character.


Input: Hello Fantastic World!
Expression: UCase( [FieldName] )
Output: HELLO FANTASTIC WORLD!
Explanation: Returns the entire string in upper case.


Input:
Hello Fantastic World!
Expression: LCase( [FieldName] )
Output: hello fantastic world!
Explanation: Returns the entire string in lower case.

Tuesday, November 17, 2009

Color Brewer 2.0 – Ideal Color Schemes for Cartographic Products

Here is an extremely useful resource for selecting color schemes for your cartographic products. It’s a website called ColorBrewer 2.0, developed by Cynthia Brewer, professor of Geography at Penn State.

Color Brewer 2.0

Color schemes can be selected for different classification types (sequential, diverging, and qualitative) and for any number of classes (ranging from three to twelve). The website provides the numeric values for colors in RGB, CMYK, and HEX, and provides options to show only colors that are colorblind safe, photocopy-able, and print friendly. The graphic below is a screenshot of the website which portrays the user-friendly interface and the numerous options for color schemes and data representation.

Friday, October 30, 2009

Find and Replace for Geoprocessing

In my last post I explained how to replace values in an attribute table using the Find & Replace... tool that is available under the Options button of the attribute table. This works very nicely for replacing characters in a specified field but unfortunately does not exist as a geoprocessing tool, which means it cannot be used in Model Builder.

I have updated the Batch Edit String Fields toolbox that I blogged about in a previous entry. I have replaced the script formerly called Replace with a script called Find and Replace. This new Find and Replace script searches a field in an attribute table for user defined characters entered in the script GUI, and replaces them with characters, also entered in the script GUI. The script can be drug into Model Builder and be run as a geoprocessing tool.



The new script has been added to the Batch Edit String Fields toolbox (see Batch Edit String Fields - Free Tools for Calculating String Fields for more information on the scripts in this toolbox). You can download the toolbox from the geospatial database. Right click the link and select Save Target As... Unzip the folder and be sure to store the toolbox in the same location as all of the Python scripts – relative paths are used to locate the scripts.

Batch Edit String Fields