Sunday, February 21, 2010

Using Python to Generate Status Emails for Lengthy Geoprocessing Tasks

In honor of PyCon (Python conference in downtown Atlanta) this weekend, I've decided to dedicate this post to a very valuable Python capability that can be extremely useful when running daily, behind-the-scenes scripts or scripts with lengthy processing times.

If there is a script that you run daily for some process such as an overnight synchronization, or an overnight cache, it can be a hassle checking the script's log file every morning to make sure it ran successfully. Or if there is a script you are running on an extremely large dataset with hundreds of thousands, or even millions of features, it would be nice to receive an email when the script completed that contained a subject providing information on whether the script completed successfully or stopped running due to some error, and in the message body see the error that caused the script to fail.

This can be done very easily with Python. I have uploaded a script to the Geospatial Database called PyEmail.py and it can be downloaded by right clicking on the link below and selecting "Save Target as...".

PyEmail.py

The uploaded script is written to work so that an email is generated once the script finishes running. If the script runs successfully, the email's subject displays "Script Completed Successfully!" and the message body is left blank. However, if the script fails, the subject displays "Script Failed", a log file with error messages is attached to the email, and the message body of the email notifies the user to check the attached log files for errors. The script can be easily modified to meet your own personal needs - maybe you only want to receive an email if the script fails, maybe you want the error messages to just print to the message body instead of having a log file, etc. All of these things are pretty simple changes within the script.

What the script requires from you:
1) Your own code which you will paste into the PyEmail script.
2) The path to your log file (if you are using a log file to catch error messages).
3) The email address(es) of the recipient(s).
4) The name and email address of the sender.
5) IP or name of the mail server used to send the email.
6) SMTP user name and password (if you are using SMTP authentification).

12 comments:

  1. dude,
    this blog is great. I just ran across it. Thanks for taking the time to put this stuff up there.

    ReplyDelete
  2. Awesome post and script. We are on a Google Apps domain and I had to make the following changes:

    # IP or name of mail server
    mailServer = smtplib.SMTP('smtp.gmail.com', 587)
    if AUTHREQUIRED:
    mailServer.ehlo()
    mailServer.starttls()
    mailServer.ehlo()
    mailServer.login(smtpuser, smtppass)

    Thank you for putting this together.

    ReplyDelete
  3. Brooke, you have no idea how helpful this script is. we have automated just about everything here via SQL jobs and this is the handiest thing I have ever seen. One qusetion for you as I am still fairly new to Python....we just came across a Python script where the positive results need to be sent to one group and the failed results need to be sent to another. I have tried numerous variations but cannot seem to make this work. Anyway you can help with this one? I have sample code and errors of what I have tried for anyone's review.

    Thanks,
    Michelle

    ReplyDelete
  4. Hi Michelle,

    I'm glad you have found this script useful. What format are the "positive results" and "failed results" in? What should be sent in the email? I just need a little more information.

    Brooke

    ReplyDelete
  5. The script bascially runs as such - if the script is successful, it should send an email to Email1@ftw.gov & Email2@ftw.gov. If the script fails, it should send an email to Email1@ftw.gov.

    This was my attempt at what I am trying to do:
    # Set up output directory for logfile
    outdir1 = "\\\\fs100\\gisuser\\Projects\\Snap_Import_Table"
    outdir2 = "\\\\GS004\\GisLogFiles\\SNAP"

    if status == 1: # If script completed successfully
    # Set up subject
    to = ["zz_IT_Basemap@fortworthtexas.gov, Randy.Westerman@fortworthtexas.gov"]
    subject = "The SNAP Import Script Completed Successfully"
    mailtext = "The SNAP data import & update was successfully completed. You can open the most recent Geocode Table at " + outdir1 + " to review & save the address list. \n"
    message.attach( MIMEText(mailtext) )
    else: # If script failed
    # Set up subject and body
    to = "zz_IT_Basemap@FortWorthTexas.gov"
    subject = "The SNAP Import Script Failed"
    mailtext = "The SNAP data import & update has failed. Please open the most recent logfile at " + outdir2 + " to review the errors. \n"
    message.attach( MIMEText(mailtext) )

    # Set SMTP Authentifiction
    AUTHREQUIRED = 0 # if you need to use SMTP Authentification set to 1
    #smtpuser = 'smtp.user@domain.com'
    #smtppass = 'smtppassword'

    #for recipient in recipients:
    # Set up from/to
    sender = "SQLGIS@fortworthtexas.gov, "
    #to = recipients

    # Set up email
    message["From"] = sender
    message["To"] = to
    message['Date'] = formatdate(localtime=True)
    message["Subject"] = subject

    # IP or name of mail server
    mailServer = smtplib.SMTP('smtp.cfwmail.com')
    if AUTHREQUIRED:
    mailServer.login(smtpuser, smtppass)
    # Send e-mail
    mailServer.sendmail(sender, to, message.as_string())
    mailServer.quit()

    When I run this though, my errors are not so much running back to the "SNAP" script as we can call it, but the errors are in the email modules from Python/ArcGIS10, etc.

    Stuff like this:
    File "\\GS005\c$\ArcGIS\Custom\PythonScripts\SNAP_Import.py", line 240, in main() File "\\GS005\c$\ArcGIS\Custom\PythonScripts\SNAP_Import.py", line 237, in main mail(flag) File "\\GS005\c$\ArcGIS\Custom\PythonScripts\SNAP_Import.py", line 82, in mail mailServer.sendmail(sender, to, message.as_string()) File "C:\Python26\ArcGIS10.0\lib\email\message.py", line 135, in as_string g.flatten(self, unixfrom=unixfrom) File "C:\Python26\ArcGIS10.0\lib\email\generator.py", line 84, in flatten self._write(msg) File "C:\Python26\ArcGIS10.0\lib\email\generator.py", line 116, in _write self._write_headers(msg) File "C:\Python26\ArcGIS10.0\lib\email\generator.py", line 162, in _write_headers header_name=h, continuation_ws='\t').encode() File "C:\Python26\ArcGIS10.0\lib\email\header.py", line 403, i...

    ReplyDelete
  6. Okay. Do you get the same error when the script fails? I think your error may be caused by the way that you have the variable "to" defined when the script is successful. Try this:

    Under the condition for if your email is successful, set the "to" variable like this:
    to = "zz_IT_Basemap@fortworthtexas.gov;Randy.Westerman@fortworthtexas.gov"

    Then when you call the sendmail method, change it to this:
    server.sendmail(sender, to.split(";"), message.as_string())

    Let me know how it goes.

    ReplyDelete
  7. I don't get the same error when it fails. It seems to like the single email mode. Modified the code as requested, less resistance, but got this:

    Traceback (most recent call last):
    File "C:\ArcGIS\custom\PythonScripts\SNAP_Import.py", line 240, in
    main()
    File "C:\ArcGIS\custom\PythonScripts\SNAP_Import.py", line 237, in main
    mail(flag)
    File "C:\ArcGIS\custom\PythonScripts\SNAP_Import.py", line 82, in mail
    mailServer.sendmail(sender, to.split(";"), message.as_string())
    AttributeError: 'list' object has no attribute 'split'

    Then I tried this after your suggestions and a bit of research and it works. It might be a little messy, but it succeeds nonetheless:

    # Set up the email module
    def mail(status):

    # Initialize message
    message = MIMEMultipart()

    # Set up output directory for logfile
    outdir1 = "\\\\fs100\\gisuser\\Projects\\Snap_Import_Table"
    outdir2 = "\\\\GS004\\GisLogFiles\\SNAP"

    # Set SMTP Authentifiction
    AUTHREQUIRED = 0 # if you need to use SMTP Authentification set to 1
    #smtpuser = 'smtp.user@domain.com'
    #smtppass = 'smtppassword'

    #for recipient in recipients:
    # Set up from/to
    sender = "SQLGIS@fortworthtexas.gov, "
    #to = recipients

    if status == 1: # If script completed successfully
    # Set up subject
    to = ["zz_IT_Basemap@fortworthtexas.gov; Randy.Westerman@fortworthtexas.gov"]
    subject = "The SNAP Import Script Completed Successfully"
    mailtext = "The SNAP data import & update was successfully completed. You can open the most recent Geocode Table at " + outdir1 + " to review & save the address list. \n"
    message.attach( MIMEText(mailtext) )
    # Set up email
    message["From"] = sender
    message["To"] = ";".join(to)
    message['Date'] = formatdate(localtime=True)
    message["Subject"] = subject

    else: # If script failed
    # Set up subject and body
    to = "zz_IT_Basemap@fortworthtexas.gov"
    subject = "The SNAP Import Script Failed"
    mailtext = "The SNAP data import & update has failed. Please open the most recent logfile at " + outdir2 + " to review the errors. \n"
    message.attach( MIMEText(mailtext) )
    # Set up email
    message["From"] = sender
    message["To"] = to
    message['Date'] = formatdate(localtime=True)
    message["Subject"] = subject

    # IP or name of mail server
    mailServer = smtplib.SMTP('smtp.cfwmail.com')
    if AUTHREQUIRED:
    mailServer.login(smtpuser, smtppass)
    # Send e-mail
    mailServer.sendmail(sender, to, message.as_string())
    mailServer.quit()


    If you have any other suggestiong on the code, please feel free to comment - that goes for anyone, but thank you very much Brooke for helping me get this up and running. I am learning quite a bit - Keep the Blog up!

    ReplyDelete
  8. The error suggests that you might have left the list brackets around the email addresses in "to". Make sure you remove the brackets so that Python doesn't think it's a list. It should just be a string:
    to = "zz_IT_Basemap@fortworthtexas.gov;Randy.Westerman@fortworthtexas.gov"

    ReplyDelete
  9. And that just shows you how new I am at this. Thanks - I put your original suggestion back into the original code and modified the brackets and it works like a charm! Thanks again Brooke!

    ReplyDelete
  10. No problem! I'm glad you got it working.

    ReplyDelete
  11. How would I go about turning off the email if the script is successful and only email when it failed?

    ReplyDelete
  12. In this article we examine the cross cultural differences with relation to status and analyse how they manifest in certain areas in the workplace. For the sake of simplicity we identify two types of status; 'ascribed-status' and 'achieved-status'. Status

    ReplyDelete