Thursday, December 24, 2009

Reverse Geocoding with Google Maps

Here's a handy link for reverse geocoding. The service is provided by Digital Inspiration and uses Google Maps to return the address at the location of a single left mouse click. One thing to keep in mind: because the reverse geocoder uses the Google Maps API, the service is only as accurate as the street data in Google Maps.

Thursday, December 10, 2009

String Functions for ArcMap Field Calculator

Back in August I posted an entry on a suite of Python scripts that I wrote for altering strings in the ArcMap attribute table. This included scripts for capitalizing words, converting lower case to upper case and vice versa, trimming white spaces, and stripping characters from the beginning and end of strings. Click here to read that entry titled Batch Edit String Fields – Free Tools for Calculating String Fields and to download the toolbox and the corresponding scripts.

While the Python scripts have been very popular among followers of Geospatial Analyst, I’ve noticed that a lot of the traffic on my blog originates from people searching for the correct VB syntax to alter string attributes. Many people are just looking for the syntax to quickly alter a string field, instead of downloading and using the Python scripts that I’ve made available. So I’ve decided to write this entry on VB field calculator expressions for common string manipulations.

***The first thing to note: Many people mistakenly search for the syntax for LTrim() and RTrim() functions when they want to strip characters from the beginning or end of a string. However, the LTrim() and RTrim() functions simply remove preceding and trailing white spaces, respectively, from the beginning and end of the string. To strip characters (instead of white spaces) from a string, you must use the Right(), Left(), or Mid() functions.***

Listed below are expressions to plug into the field calculator along with the resulting output for the string “Hello Fantastic World!”.



Input: Hello Fantastic World!
Expression: Left( [FieldName], 5)
Output: Hello
Explanation: Returns the first 5 characters from the left.


Input: Hello Fantastic World!
Expression: Left( [FieldName], Len( [FieldName] ) -17)
Output: Hello
Explanation: Returns the string minus the last 17 characters.


Input:
Hello Fantastic World!
Expression: Right( [FieldName], 16)
Output: Fantastic World!
Explanation: Returns the last 16 characters from the right.


Input: Hello Fantastic World!
Expression: Right( [FieldName], Len( [FieldName]) -6)
Output: Fantastic World!
Explanation: Returns the string minus the first 6 characters.


Input: Hello Fantastic World!
Expression: Replace( [FieldName], "Fantastic", "Beautiful")
Output: Hello Beautiful World!
Explanation: Replaces the first string with the second string.


Input:
Hello Fantastic World!
Expression: Mid( [FieldName], 7, 9)
Output: Fantastic
Explanation: Returns the first nine characters beginning at the 7th character.


Input:
Hello Fantastic World!
Expression: Mid( [FieldName], 17, 5)
Output: World
Explanation: Returns the first 5 characters starting at the 17th character.


Input: Hello Fantastic World!
Expression: UCase( [FieldName] )
Output: HELLO FANTASTIC WORLD!
Explanation: Returns the entire string in upper case.


Input:
Hello Fantastic World!
Expression: LCase( [FieldName] )
Output: hello fantastic world!
Explanation: Returns the entire string in lower case.

Tuesday, November 17, 2009

Color Brewer 2.0 – Ideal Color Schemes for Cartographic Products

Here is an extremely useful resource for selecting color schemes for your cartographic products. It’s a website called ColorBrewer 2.0, developed by Cynthia Brewer, professor of Geography at Penn State.

Color Brewer 2.0

Color schemes can be selected for different classification types (sequential, diverging, and qualitative) and for any number of classes (ranging from three to twelve). The website provides the numeric values for colors in RGB, CMYK, and HEX, and provides options to show only colors that are colorblind safe, photocopy-able, and print friendly. The graphic below is a screenshot of the website which portrays the user-friendly interface and the numerous options for color schemes and data representation.

Friday, October 30, 2009

Find and Replace for Geoprocessing

In my last post I explained how to replace values in an attribute table using the Find & Replace... tool that is available under the Options button of the attribute table. This works very nicely for replacing characters in a specified field but unfortunately does not exist as a geoprocessing tool, which means it cannot be used in Model Builder.

I have updated the Batch Edit String Fields toolbox that I blogged about in a previous entry. I have replaced the script formerly called Replace with a script called Find and Replace. This new Find and Replace script searches a field in an attribute table for user defined characters entered in the script GUI, and replaces them with characters, also entered in the script GUI. The script can be drug into Model Builder and be run as a geoprocessing tool.



The new script has been added to the Batch Edit String Fields toolbox (see Batch Edit String Fields - Free Tools for Calculating String Fields for more information on the scripts in this toolbox). You can download the toolbox from the geospatial database. Right click the link and select Save Target As... Unzip the folder and be sure to store the toolbox in the same location as all of the Python scripts – relative paths are used to locate the scripts.

Batch Edit String Fields

Thursday, October 29, 2009

Find and Replace

This post is a response to a question from Ivar that was posted under the Batch Edit String Fields blog entry. The question was:

Brooke, I'm looking for a script that deletes or replaces some characters from string values. I have one field where the string values will be used as names for feature classes when I split the feature class into many, one for each feature. Example: I would like to replace the text "1630089 - 1630" with "16300891630" or "1630089_1630" (I'm not sure which one will work). It is the " - " part of the string that is the problem. I see that I might find a sollution by using several of the scripts posted by you, but if there is one script that will do the job, I'm grateful for directions.

Thanks!
Ivar


Replacing the text in a string field can be accomplished with the Find and Replace tool available in core ArcGIS Desktop at all license levels. To do this:

1) Start editing.
2) Open the attribute table.
3) Click on the field heading containing the field text you want to replace to ensure only that field will be altered.
4) Click the Options button at the bottom of the attribute table and choose Find and Replace.
4) A dialog box opens with two tabs. Select the Replace tab. Choose to find " - " (without the quotes). If you want to replace it with an under score (like 1630089_1630), enter "_" (without the quotes) in the Replace input box. If you want to combine the text on either side of the " - " (like 16300891630) simply leave the Replace input box blank.



5) Check on the Search Only Selected Field(s) option and select to match Any Part of the text.
6) Select Replace to replace the values one at a time (might be good to do this for the first couple to make sure you are getting the desired result) or select Replace All to replace all of the values at once.
7) Save edits and stop editing.

Thanks for reading my blog and posting your question, Ivar. I hope this does what you are looking to accomplish.

Wednesday, September 30, 2009

How to Convert ArcINFO Interchange Files (.e00) to Shapefiles (.shp)

Back in the days of ArcINFO, interchange files, also known as export files (.e00) were used to package directory based data types (coverages and grids) into a single file so they could be transferred between different machines. Though ArcINFO interchange files are used less frequently today, it is not uncommon to come across this file type. So the question is, how do you convert an export file (.e00) to shapefile (.shp) format? There are actually a couple ways. The easiest way is to use ArcGIS Desktop (ArcView, ArcEditor, or ArcInfo license level). I explain how to do this step-by-step below. However, if you do not have ArcGIS Desktop, you can install a couple of free tools to achieve the same results. See the Converting E00 Vector Data To Shapefiles – A Free and Fairly Painless Approach entry on the Free Geography Tools blog for information on and links to the free tools available for the conversion.

Conversion Using ArcGIS Desktop
Stage 1: Convert ArcINFO interchange file to ArcINFO coverage
1. In ArcCatalog, click the View menu, then click Toolbars, and select ArcView 8x Tools to open the toolbar.
2. Select the Conversion Tools drop-down menu and click on Import from Interchange File…
3. Click the Input file browse button and navigate to the .e00 file to be imported. Once the .e00 file is selected click Open.
4. Click the Output dataset browse button and navigate to the directory where the output coverage will be stored. Provide a name for the coverage and click Save.
5. Click OK.



Stage 2: Convert ArcINFO coverage to Shapefile
6. In ArcToolbox, navigate to the Feature Class to Shapefile tool located in Conversion Tools  To Shapefile.
7. Click the Input Features browse button and navigate to the ArcINFO coverage that was created in Stage 1. Select the coverage and click Add.
8. Click the Output Folder browse button and select the folder where the shapefile will be stored. Click Add.
9. Click OK.



*Important Notes*
• The input .e00 cannot have a space in its name or be stored in a directory that has a space anywhere in the directory’s path.
• The output coverage cannot have a space in its name or be stored in a directory that has a space anywhere in the directory’s path.
• The output coverage cannot be more than 13 characters long, and cannot contain special characters (underscores are okay).

Monday, September 21, 2009

Hawth’s Tools – Free Tools for Spatial Analysis

Hawth’s Analysis Tools provides a suite of solutions for tasks common in spatial analysis. Specifically developed for ecological studies, these tools can be used in any application for analyzing spatial data. They extend (and in some cases simplify) core ArcGIS functionality that is not available out-of-the-box. The extension consists of more than fifty tools that cover a broad range of analysis types including analyzing, sampling, and editing vector as well as raster data, and tools for common operations in tables and CSV files. All tools and descriptions can be found here:

Tool Descriptions

A few of the tools that I have found useful in my work include:
• Create Random Selection tool for selecting specified number or percentage of random features (points, lines, or polygons)
• Generate Random Points tool for generating a defined number of points over a specified area
• Create Vector Grid for generating a grid over a specified area and of a defined grid cell size
• Thematic Raster Summary for summarizing the frequency of cells of categories in a thematic raster layer by polygon
• Sum Values for calculating the total value of all/selected rows in a numeric field
• List Unique Values for listing all unique values in a selected field
• Delete Multiple fields for removing multiple fields from a table all at once
• Intersect Lines for generating point features at line intersections

The tools run on ArcGIS 9.x at the ArcView, ArcEditor, and ArcInfo license levels. *These tools do not support on-the-fly projection changes, so it is important that tools that use more than one input layer have the same projection.

Thursday, September 10, 2009

Online Batch Geocoding – Address Locators and Street Networks A Thing Of The Past

Here’s a handy website that will leave you wondering why you ever dealt with standardizing addresses, building address locators, or trying to find a decent street network for geocoding:

www.batchgeocode.com

The website uses the Yahoo! Geocoding API to locate street addresses and convert them to latitude and longitude coordinates in decimal degrees using the WGS84 coordinate system. Street level geocoding is available for the United States, Canada, and many European countries (see the Data Coverage section on the website for a complete list). The geocoder runs relatively fast and generates accurate results.

How to use:
Ensure that your address data is broken into four fields:
1. Address (house number, street name, and street type)
2. City
3. State/Province or Country (Europe)
4. Zip/Postal Code

If your data is in an Excel spreadsheet, select all of the cells, click copy, and then paste the contents in the space provided on the website (under step #2). Follow the easy instructions and click Run Geocoder. Once it’s done running, two columns are added to the table for latitude and longitude (bg_lat and bg_long). The results are displayed in a text box under Step #6. Follow the instructions for getting this data back into Excel.

Once you have your data back into Excel, you can easily create point features for your addresses. Ensure the new spreadsheet containing the fields for latitude and longitude is saved. In ArcMap 9.x (with ArcView, ArcEditor, or ArcInfo licenses), select Add XY Data… under the Tools menu.



Locate the Excel spreadsheet and set the X and Y values to the appropriate fields in the table. Select an appropriate coordinate system and click OK.



An event layer containing the point features of your addresses are added to the Table of Contents. *To store this layer as a shapefile or feature class, you must export the features in the event layer.* Do this by right clicking the layer in the Table of contents and selecting Data  Export Data…



Navigate to the appropriate directory for storing the data and click OK.

Monday, August 31, 2009

Free GIS Data for Washington, D.C.

Whether you are working on a GIS project for a class assignment or setting up a demo to convey functionality in ArcMap, the biggest challenge is usually finding (free) spatial data. Even if you are able to find the data required to present the main theme of your project/demo, it is often difficult to find the surrounds necessary for a cartographic display of the results.

One great source for spatial data is the DC GIS Data Inventory provided by the Washington, D.C. local government. Their website contains hundreds of datasets from A to Z (literally) in both shapefile and keyhole markup language formats. Data exists for a broad range of categories for the entire District. I have listed only a fraction of available data below to point out some of the most common/useful datasets, but you can visit the DC GIS Data Inventory website for a complete directory.

Transportation
• Streets
• Bike Routes
• Bus Routes
• Bus Stops
• Metro Lines
• Metro Stops
• Sidewalks

Environmental
• Soil Polygons
• Wetlands
• Wooded Areas
• Watersheds
• Air Emissions

Infrastructure
• Utility Poles
• Electrical Substations
• Fire Hydrants
• Buildings

Commercial
• Retail Sites
• Gas Stations

Surrounds
• City Boundary
• Census Boundaries
• Streets
• Railroads
• Parks
• Water Bodies

The above is merely a snapshot of what is available on the site. You could easily create a mulit-modal transportation network, complete an environmental research project, present a detailed cartographic display of your project/demo results, and much more. Additionally, each dataset contains highly detailed metadata in Federal Geographic Data Committee (FGDC) and ESRI standards. The metadata for each dataset can be viewed prior to downloading the data.

So if you have a project idea for a class assignment or demo that isn’t location specific, establishing Washington, D.C. as the setting for your analysis may be a pliable option given the broad range of available data.

Tuesday, August 18, 2009

Get XYZM - Populate X, Y, Z, and M Attributes

So here it is, as promised - a tool for populating attributes with values for the X, Y, Z, and/or M dimensions of point features. It is compatible with ArcGIS Desktop 9.x at the ArcView, ArcEditor, and ArcInfo license levels. Download it for free here (right click and select Save Target As...):

Get XYZM

The zip file contains the tool (GetXYZM.dll) and a read me file (readme.txt). Unzip the folder to your hard drive. To use the Get XYZM tool, open ArcCatalog and select Customize from the Tools menu. Under the Commands tab, click Add from file… and navigate to the directory containing the unzipped files.



Select the GetXYZM.dll and click Open. Click OK on the Added Objects form. The tool is now listed under the Commands window. Select the Get XYZM command and drag it to the toolbar. You can create a new toolbar or add it to an existing toolbar. Click Close in the Customize dialogue box.



To activate the tool, you must select a point shapefile or feature class in the Catalog tree. Select the point data to populate with dimension values and click the Get XYZM tool button that you just added to the toolbar to bring up the Get XYZM dialogue box.



If your point data already contains fields to populate, ensure the Use existing fields option is selected and choose the appropriate fields in the dropdown menu for the corresponding dimension. You can choose to populate any combination of dimensions – you don’t need to provide a field for each. Once the existing fields for dimension values to populate are selected, click OK. The dialogue box closes automatically when the tool is finished running.



If your point data does not contain existing fields for storing the values of the desired dimensions, select the Create new fields option and enter the field names in the corresponding text boxes provided for each dimension. Once the desired field names to add and populate are entered, click OK. The dialogue box closes automatically when the tool is finished running.



After you select/enter the appropriate fields and click OK - VoilĂ ! - the fields are populated with the chosen dimensions!

Thursday, August 13, 2009

Easy Calculate 5.0 - Free Tools for Calculating Shape Geometries

I'm writing this entry in response to StuGotz's comment on my last post (Batch Edit String Fields - Free Tools for Calculating String Fields). StuGotz currently uses VB to populate fields with X, Y, Z, and M values but is looking for a more simplistic method for calculating these fields. This is a good opportunity for a post on Easy Calcuate 5.0 - a set of files (.cal) containing code for calculating a wide range of geometric properties for vector data (points, lines, and polygons) that can be easily loaded into the Field Calculator. The files are available for free here:

Easy Calculate 5.0

Just download the zip file and unzip the file to a directory on your hard drive. These .cal files are extremely easy to use and come in very handy for common geospatial calculations including the very properties that StuGotz is looking for - the X, Y, Z, and M dimensions for point features. Also included are files for calculating the from, to, and midpoint X and Y coordinates of a line, the maximum and minimum X, Y, Z, and M values for points, multipoints, and polygons, the X and Y coordinates for the center of polygons, the thickness ratio of each polygon, and many, many more (see above link for a complete list).

Okay, so I mentioned Easy Calculate 5.0 is easy to use, right? I didn't lie. Once you've downloaded and unzipped the file, here's all you have to do:

1. Right click on a field in the attribute table in ArcMap and select Calculate Values.
2. In the Field Calculator dialogue, click the Load button, navigate to the directory where you unzipped the files, and select the appropriate .cal file for your calculation.
3. Make any changes to the code *if needed* (adjust tolerance, etc).
4. Click OK.

Now is that easy or is that easy? I can see why they called it "Easy Calculate"!

In addition to geospatial calculations on attribute fields, Easy Calculate 5.0 also provides methods for editing the actual geometry of vector features including offsetting lines by a specified distance, generalizing lines and polygons using the Douglas Poiker algorithm, converting polygons to their bounding rectangles (envelopes), reversing the direction of lines, and more. Simply run the Field Calculator on the Shape field when using the files for editing feature geometry.

Field calculations using these files can be made at any of the ArcGIS license levels (ArcView, ArcEditor, and ArcInfo) with ArcGIS 8.x and 9.x. Changes to the fields and feature geometry are applied directly to the original data. If the .cal files are applied out of an Edit session (not available with ArcView), it is a good idea to make a copy of the original data, as the changes are irreversible. Calculations and geometry edits are applied only to selected features. If no features are selected, values for all records will be calculated.

Now, it's not Christmas or anything but I was feeling pretty generous and decided to work on a custom tool for StuGotz so that the fields for X, Y, Z, and M values can be populated with the click of a button (as if Easy Calculate weren't easy enough). So stay tuned and subscribe and/or follow this blog. Once I finish the tool, I'll write an entry on how to use it and upload it to the geospatial database so that it can be downloaded for free.

Saturday, August 8, 2009

Batch Edit String Fields - Free Tools for Calculating String Fields

If you’ve ever tried to make batch edits on all or a selection of string values in an attribute table, you’ve probably spent a significant amount of time searching for the correct VBScript to plug into the Field Calculator dialogue box to get the desired results. Well search no more! A few years back I wrote the following Python scripts to simplify common batch edits made to string fields:

  • Capitalize: Capitalizes the first letter of a string. Ex. Changes "hello world" to "Hello world"
  • Capitalize Words: Capitalizes the first letter of every word in a string. Ex. Changes "hello world" to "Hello World"
  • Lower Case to Upper Case: Converts lower case letters of a string to upper case. Ex. Changes "hello world" to "HELLO WORLD"
  • LStrip: Strips a specified number of preceding characters of a string.
  • LTrim: Removes preceding white spaces from the end of a string.
  • Replace: Replaces current string with a new string.
  • RStrip: Strips a specified number of trailing characters of a string.
  • RTrim: Removes trailing white spaces from the end of a string.
  • Upper Case to Lower Case: Converts upper case letters to lower case. Ex. Changes "HELLO WORLD" to "hello world"

Download the toolbox with all nine Python scripts for free! The link below retrieves a zip file containing a toolbox (.tbx), nine Python scripts (.py) listed above, and a readme.txt from the geospatial database. Right click the link and select Save Target As... Unzip the folder and be sure to store the toolbox in the same location as all of the Python scripts – relative paths are used to locate the scripts.

Batch Edit String Fields

Minimum Requirements:
ArcView license of ArcGIS 9.2
Python 2.4

The scripts can be run in either ArcMap or ArcCatalog. Edits can be made on all records in a feature layer, feature class, or shapefile. If a selection is used, the scripts will only run on the selected records. A SQL expression can be applied to the script to run on only those features that meet the conditions of the SQL expression.

*Important Note*
All changes made using these scripts are permanent unless they are used in an Edit session (ArcEditor or ArcInfo license required).





In this example, the Capitalize Words tool converts the first letter in each word in the ADDRESS field from lower to upper case.

Tuesday, August 4, 2009

What is GIS?

Okay so here’s a good topic to start my blog with: What is GIS? How do you really define and explain it to someone who doesn’t know what it is?

As a Geospatial Analyst I know the ins and outs of GIS, as I use it in my job every day. Though I’m not so sure you’d get that from me if you heard me try to explain what GIS is to someone who has never heard of it before. Instead of giving a brief, understandable definition of GIS, I find myself painfully stuttering through what ends up becoming a boring, broken monologue to which I eventually receive a blank stare. This is the point at which I usually just awkwardly stop talking, as I end with “Yeah…it’s just kinda hard to explain”.

Here’s my problem: GIS is so broad of a topic and so powerful of a tool, I want people to understand and RECOGNIZE its capabilities. So instead of explaining it at a high level, I find myself overwhelmed trying to figure out how I can briefly explain all of the capabilities and applications of GIS to someone who probably doesn’t know much about databases in general – let alone spatial databases. After talking to my fellow GIS geek friends, this seems to be a pretty common problem among all of us in the GIS field – explaining what it is that we do every day. So, I’ve put together some thoughts on best practices for approaching this topic and explaining what GIS is at a high level.

First tip: You’re never going to be able to briefly explain GIS to someone in a way that they will be able to understand its power and applications if they have never used it before – don’t waste your time or theirs with details.

Okay, so what IS GIS anyway? *At a high level*, GIS is a combination of a few things:
1. Hardware: workstations, servers, and handheld devices for collecting and storing spatial data.
2. Software: GIS application for viewing, analyzing, and transforming spatial data.
3. Spatial data: data with a geographic location.
4. People: analysts well-trained in spatial analysis and skilled in using GIS software.

Let’s practice. Someone asks you, “What is GIS?”. Instead of stuttering your way through the details, you calmly and collectively say:

“GIS stands for Geographic Information Systems. It’s a computer-based system for collecting, viewing, analyzing, storing, and transforming spatial data, or data with a geographic location.”

You might want to give a *few* descriptive examples, so that people have a better idea of what you are talking about. So here are some applications of GIS with examples that you may want to mention:

  • Commercial: Use spatial models and geodemographics to target markets and improve profitability.
  • Law Enforcement: Use incident mapping and spatial statistics to analyze crime.
  • Transportation: Collect roadway pavement conditions to determine funding and plan pavement schedules.
  • Utilities: Model electric networks to determine customers effected by power outages.
  • Logistics Management: Vehicle routing for school buses, delivery services, emergency vehicles, etc.
  • Environmental: Locate ideal habitat locations for studying wildlife.

So there you have it – my very own simplified explanation of GIS. Post a comment or shoot me an email if you have another definition. I could throw a list of good definitions together for a later blog.

Friday, July 31, 2009

Welcome!

Hi! Welcome to my blog - your one-stop shop for all things geospatial; a "geoportal" for sharing geospatial news, ideas, thoughts, opinions, tips and tricks, and FREE GIS tools and data. This blog is intended for all geospatial analysts, and is specifically targeted towards every-day GIS users around the globe. Visit my blog to find and share helpful Python scripts and bits of ArcObjects code to streamline and simplify repetitive and complex tasks, learn about free downloadable third party tools, software, and data, and read about hot topics and books in Geography. Feedback and comments are welcomed and encouraged!

Stay tuned for new entries...