Wednesday, December 1, 2010

From/To Replace String - Another Addition to the Batch Edit String Fields Toolbox

From/To String Replace is the latest script added to the Batch Edit String Fields Toolbox (see the Batch Edit String Fields - Free Tools for Calculating String Fields entry). A coworker needed to edit a field for labeling River polygons on a cartographic product in ArcMap. The field was a Name field but contained extra unwanted verbiage in parentheses to further explain the Name. For example:

New River (North Carolina portion)

and

Roanoke River (Lake Gaston below normal full power pool elevation 200 MSL)

She only wanted the labeling to show the river name, and not the explanation contained in parentheses. The Find and Replace script in the Batch Edit String Fields Toolbox is handy for replacing known substrings in a text field, but in this case, anything could be within the parentheses, so using the Find and Replace script wouldn’t work. [Enter the new From/To String Replace script!]

See the image of the script below. With this script, the starting character as well as the ending character of the substring you’d like to replace/remove is taken into account. The Replace With parameter is for the new string that will replace the characters to be removed. If there is nothing to replace the removed string with (like in the example above), the Replace With parameter can be left blank, and the substring will simply be removed from the field.

The From/To String Replace script can be downloaded as part of the Batch Edit String Fields Toolbox from the Geospatial Database. Feel free to provide feedback if you found this useful or if you have ideas for further enhancements.

Batch Edit String Fields Toolbox

*Note: These scripts were written in ArcGIS Desktop 9.3 and 9.3.1. They will work in ArcGIS Desktop 9.2. 9.3, 9.3.1, and 10.0 at the ArcView, ArcEditor, and ArcInfo license levels. Python 2.4, 2.5, or 2.6 is also required.

Sunday, November 21, 2010

Stressful day at the office? Not for us...

Check this out. A friend sent me this link about a month and a half ago. CNNMoney.com and Payscale.com conducted a survey on more than 40,000 workers and compiled a list of the “10 Least Stressful Jobs”. Geographic Information Systems Analyst is number five on the list!

CNNMoney.com/Payscale.com Survey

This got me thinking – we (GIS/Geospatial Analysts), generally speaking, really are in a great field for so many reasons. Our jobs are challenging and can be so rewarding. We often have the opportunity to work with domain experts in other fields, allowing us to learn about other industries. Because the technology is always advancing, it’s essential to stay on top of the current, cutting-edge solutions. Moreover, it just seems like people who work in GIS are so passionate and excited about the work that they do. Positive attitudes coupled with low stress… now THAT makes it easier to get up and go to work every day. We’ve got it good.

Tuesday, June 22, 2010

Access your Oracle Database from Python

cx_Oracle is a free, downloadable Python extension module that enables Python to access an Oracle database. This can come in handy when you need to streamline or automate tasks that require the use of SQL statements to access components of your spatial database, such as feature classes and domains. The module is available for all combinations of a variety of Operating Systems, Oracle versions (9.2.0, 10.2.0, and 11.1.0), and Python versions (2.0 and up). *Note: An Oracle client (or server) installation is required in order to use cx_Oracle. If you do not require the tools that come with a full client installation, it is recommended to install the Instant Client which is far easier to install.

I found this module to be useful when deleting feature classes with domains from the database. In ArcGIS 9.3.1 and earlier versions, unused domains are left lingering in the database after feature classes have been deleted. What’s more, default domains have to be unregistered before they can be deleted. I wrote a script in Python using the cx_Oracle module to loop through all feature classes in an ArcSDE database, remove default domains from fields, delete each feature class, and then delete the domains remaining in the database. I’ve added this script to the Geospatial Database. If you would like to see an example of how to use cx_Oracle or are looking to delete feature classes with domains from a database like I described above, you can download the script here.

The script takes four arguments:
1) Owner: Data owner.
2) Password: password@NET SERVICE NAME
3) Workspace: The folder location of the sde database (and feature dataset, if applicable) containing feature classes to run the script on (ex. C:\Documents and Settings\user\Application Data\ESRI\ArcCatalog\example.sde)
4) SDE Owner: Owner of the SDE repository in the database (ex. sde)

Friday, April 30, 2010

What Is A Geospatial Analyst?

I came across this article that gives a good description of the roles of a Geospatial Analyst. I know that a lot of the traffic on my blog stems from people searching for a description of what a Geospatial Analyst does, so I thought I would share this. It mentions the skill sets required for a Geospatial Analyst position are environmental sciences, computer science, geography, and database design. I think this is a pretty inclusive list at a very high level (other than remote sensing, which is brushed upon later on in the article). Check it out here:

The Role of a Geospatial Analyst

Tuesday, March 23, 2010

"ezree"

Here’s an interesting tidbit: ESRI employees received an email from corporate last week regarding the pronunciation of ESRI. We will now be calling ourselves “ezree” instead of E-S-R-I to ensure “consistent name recognition around the world”. This makes sense as most non-ESRI employees know the industry’s leading GIS software company as “ezree”. Even I called it "ezree" prior to becoming an employee. Though now the pronunciation sort of makes me cringe. How do you feel about it?

Monday, February 22, 2010

Inserting Characters into a String - The Latest Addition to Batch Edit String Fields Toolbox

This post serves as a response to a comment posted by Josh on the Find and Replace entry. Josh commented:

"Brooke,
Let me ask a variant of Ivar's question. Ultimately I can resolve my situation the same way he identified above, however, is there a way to insert specified characters at a specific position. For instance; insert "ap" at position 2 from left for the field "Hpy" to create "Happy"? Any Ideas? Thanks. Nice site. Josh
"

Josh,
Thank you so much for your question and for visiting my blog. There are a couple ways of doing this:

1) You can use the field calculator in the layers attribute table using the following expression:
Left( [FieldName], 1) & "ap" & Right( [FieldName], Len( [FieldName]) -1)
So in this example, "ap" is added to the first character of the string (the Left function with the number 1 returns the first character), and then the characters of the original string minus the first character (the Right function with the number -1 returns all characters from the right of the original string minus the first character) are added to the end. So if the string in your field is "Hpy", "ap" will be added after the first character ("H"), and then the rest of the original string ("py") is added to the end. See the post on String functions for ArcMap Field Calculator for more VB examples for altering strings.

2) Thanks to your question, I added a new script to the Batch Edit String Fields toolbox I blogged about last September. Below is a graphic of this new tool.



The toolbox and scripts can be downloaded from the Geospatial Database. Use the link below to download the zip file. Ensure that the files are unzipped to the same folder (the scripts use relative paths).

Batch Edit String Fields.zip

Sunday, February 21, 2010

Using Python to Generate Status Emails for Lengthy Geoprocessing Tasks

In honor of PyCon (Python conference in downtown Atlanta) this weekend, I've decided to dedicate this post to a very valuable Python capability that can be extremely useful when running daily, behind-the-scenes scripts or scripts with lengthy processing times.

If there is a script that you run daily for some process such as an overnight synchronization, or an overnight cache, it can be a hassle checking the script's log file every morning to make sure it ran successfully. Or if there is a script you are running on an extremely large dataset with hundreds of thousands, or even millions of features, it would be nice to receive an email when the script completed that contained a subject providing information on whether the script completed successfully or stopped running due to some error, and in the message body see the error that caused the script to fail.

This can be done very easily with Python. I have uploaded a script to the Geospatial Database called PyEmail.py and it can be downloaded by right clicking on the link below and selecting "Save Target as...".

PyEmail.py

The uploaded script is written to work so that an email is generated once the script finishes running. If the script runs successfully, the email's subject displays "Script Completed Successfully!" and the message body is left blank. However, if the script fails, the subject displays "Script Failed", a log file with error messages is attached to the email, and the message body of the email notifies the user to check the attached log files for errors. The script can be easily modified to meet your own personal needs - maybe you only want to receive an email if the script fails, maybe you want the error messages to just print to the message body instead of having a log file, etc. All of these things are pretty simple changes within the script.

What the script requires from you:
1) Your own code which you will paste into the PyEmail script.
2) The path to your log file (if you are using a log file to catch error messages).
3) The email address(es) of the recipient(s).
4) The name and email address of the sender.
5) IP or name of the mail server used to send the email.
6) SMTP user name and password (if you are using SMTP authentification).

Wednesday, February 10, 2010

ET GeoWizards – A suite of tools extending ArcMap functionality

ET GeoWizards is a suite of GIS tools that extends ArcMap functionality and is available at all license levels. Additionally, the tools allow ArcView license holders to perform analysis tasks that would otherwise require an ArcEditor or ArcInfo license, or require the 3D Analyst extension. The full ET GeoWizards tool set without restrictions or limitations can be purchased here, however much of the functionality is absolutely free. Click here to download the tools for free (with some restrictions and data processing limitations without a license key) from ET SpatialTechniques. Installation includes a DLL that can be added to an ArcMap toolbar and provides an easy-to-use GUI with tabs of tools grouped by theme.

The installation package also includes an ET GeoWizards toolbox (.tbx) that contains tools that can be used in Model Builder and Command Line. For a complete list of free tools on both the GUI and in the toolbox, see the Free functions of ET GeoWizards page.

Below are *just a few* functions with descriptions from each tabbed group on the GUI that I personally have found useful. Definitely check out the complete list to see all of the free tools available.

Point
• Clean Point Layer – Deletes duplicate points.
• Measure Points – Assigns measures (M) or elevations (Z) to points based on interpolated values from the closest polyline within a user specified search tolerance.
• Reverse Geocoding – Uses a reference polyline layer to assign assign addresses to points of a point layer.



Polyline
• Clean Polyline Layer – Ensures topological correctness of polyline features.
• Get Z Characteristics – Calculates several Z dimension characteristics and stores them in fields added to the attribute table. Includes characteristics such as 2D and 3D length of the polyline, min and max Z values, and average and max slow uphill and downhill among others.
• Flip Polylines – Assigns the node closest to a specified start point as the from node so that the direction of all polyline features point away from the start point.



Polygon
• Clean Polygons – Ensures topological correctness of polygon features.
• Create Centerlines – Creates centerlines from polygon features. Can be created inside polygons (from polygon rivers and streets) our outside polygons (from parcel data).
• Get Adjacent Polygons – Stores the adjacent polygon names/IDs (user specified) as a comma delimited string as well as the count (number of adjacent polygons) for each polygon in the attribute table.



Convert
• Polyline to Point – Converts a polyline dataset to a point dataset at 1) vertices, 2) nodes, or 3) mid points.
• Point to Multipoint – Converts a point layer to a multipoint dataset.
• Point to Point Z (M) – Uses a numeric field to assign values to the Z or M dimensions.
• Polygon Z (M) to Point – Converts polygon vertices to points and adds attributes to store Z and/or M values of each vertex.
• Polyline Z (M) to Point – Converts polyline vertices to points and adds attributes to store Z and/or M values of each vertex.



Surface
• Build TIN – Builds a Triangulated Irregular Network from a point, polyline, or polygon feature layer.
• Features to 3D – Creates a 3D dataset from point, polyline, or polygon features by deriving the values from a surface (Raster or TIN) layer.



Geoproc
• Transfer Attributes – Transfers attributes from one polygon layer (source) to another (target) based on their spatial location.
• Spider Diagram (Spatial Location) – Creates a Spider Diagram representing the shortest distance between the points of a Center Points layer and the features in the destination layer (points, polylines, or polygons).
• Spider Diagram (Attribute Link) – Creates a Spider Diagram between the points of a Center Points layer and the features in the destination layer (points, polylines, or polygons) based on a common value in a user specified field in both layers.



Basic
• Redefine Fields – Change field names, length, precision, and scale.
• Order Fields – Exports a feature layer to a new feature class. The user selects the fields to be exported and the order in which they will appear in the attribute table.
• Vector Grid – Creates a polygon or polyline grid with user defined extents and cell size.



LinRef
• Create routes from existing polylines – Creates routes from existing polyline features. Input polylines that share a common identifier are merged to create a single route.
• Calibrate routes with points – Calibrates existing routes using the measures of existing points. The measures collected from the points can be M dimension values or can be stored in a ield of the point attribute table.
• Locate point features along routes – Finds the route and measure information of point features and creates a point event table. Search tolerance is used to correctly identify the closest routes to each point.
• Locate polygon features along routes – Calculates route and measure information at the geometric intersection of polygon data and route data and creates a line event table.



Misc
• Features To Envelopes – Creates a polygon from the envelope of each feature in the input feature class. Attributes of the original features are transferred to the envelope polygons.
• Features To Minimum Bounding Circles – Creates a circular bounding polygon from each feature in the input feature class. Attributes of the original features are transferred to the resulting polygons.
• Create Station Lines – Creates station lines along the polyline features of the input layer. The station lines are perpendicular to the original polylines with user specified station distance and length. The length can be constant or calculated from Z or M values (if the input layer has Z or M values).



In/Out
• Export To Google Earth – Converts all visible layers in a data frame to a Google Earth KML file.
• Import From Google Earth – Converts Google Earth (KML or KMZ) data in to feature classes.