a hewer of maps

Extract Invisible Attachment

2015-05-11T07:00:00Z

I periodically get email from ios users with invisible attachments.

TLDR: view raw, save base-64 to text, uudecode using certutil --decode (a native tool in Win7+)

Last night I finally figured out what is going on: the attachment is marked as "inline" in the header, but the client (gmail in this case, but can happen with others too) doesn't render the attachment correctly. However since the media is inline it's not visible/extractable like regular attachments are.

Symptoms

Email says there is in attachment (e.g. has a paperclip), but there is no place to click and save or download the attachment
broken image icon in message body

The fix

view original/raw message
Find Content-Disposition: attachment; filename="some_filename.pdf" and save everything between Content-Id: <AAA888...> and --Apple-Mail-AAA888...-- lines (don't include those lines) to a text file, e.g. some_filename.uue
open command shell and run certutil -decode some_filename.uue some_filename.pdf

Sources

Fowarded emails with attachments not showing up with paperclip icon, https://productforums.google.com/forum/#!topic/gmail/Mdujqj1zUic
Message with pdf attachment doesn't show as having an attachment, https://productforums.google.com/forum/#!topic/gmail/LbUIrEqzRXE
Super User: How can I extract an attachment from the email body (encoded as base64)
Dick on http://johnsad.ventures/software/base64encoderdecoder/ for the certutil tip

The 1st modern Cahill-Keyes Map, maybe

2013-12-19T00:26:29Z

All maps distort things -- one or more of area, shape and distance. They [splatten] a roundish sphere into a flat piece of paper. Some distortions are much worse than others. The Cahill-Keyes projection is the best invention to date we humans have for minimizing the effects. Unfortunately very few know about it or have seen it. This map by Duncan Webb of Australia, possibly the first modern full scale version of the CK map projection could go a long way to rectify our common (mis)perception of the relative shapes and sizes of where we live. If it sees production -- you can help make that happen by pledging to buy one before January 3rd:

What's all the fuss about? There are many resources out there explaining it more completely and eloquently than I can, but here's an info-graphic to get you started. Each of the pinkish blobs on the maps below occupy the same size and area of ground, but they sure don't look like it do they?

[0]: well, 'splatten' should be word darn it!

Oh Assign, Assign, wherefore art thou Assign?

2013-05-17T07:54:00Z

Most of my computer life of the last few years has lived within Dropbox. It's proven to be a very hands off largely no-thought means of ensuring the 3 Windows computers I use regularly at work and home all have almost identical tool chains and environments. It's always bugged me though, that my "home" is buried under C:\Users\matt\Dropbox (though at least with Win7 it's not under that horrible "Documents and Settings" folder!). Way too freakin much typing just to get to the point of starting work, or fun.

I've tried several methods for making this more livable, and each succeeds to a certain extent -- though none come close to matching what we had 20 years ago with the Amiga.

Move Dropbox to root of C:

yech. user data on same partition as operating system. never been comfortable with this. I want OS and Programs over [there], my docs and stuff [here], and easily wipe or restore one without touching the other.

Mapped "Network" Drive

Open a command shell, fire off net use B: \\localhost\c$\users\%username%\Dropbox /persistent:yes.

(So nice that A: and B: almost never exist anymore, and are thus available, even when joined to a domain.)

Worked swimmingly well for several months, until learning that the mysterious Mecurial and Bazaar lock errors I would encounter at seemingly random occasions and took hours to get out of were due to timing issues in the underlying network transport protocols, even though everything was happening on the local machine and darn fast, to human perception.

Also can't use any commands requiring elevated privileges because "B:" isn't available to Administrator. (why o why doesn't Windows have sudo??)

Subst-tituted Drive

subst b: c:\users\%username%\dropbox

An effective cure for the timing issues noted previously. Suffers from lack of sudo, which has been real problem while trying to work on improving Leo and Osgeo4w install packages.

Separate Partition

The Dropbox app steadfastly refuses to use a bare drive as it's home, and insists on creating a folder called Dropbox where you point it. If you run a client older than 1.6 or so, before the database was encrypted, it is possible to open the config.db file in an sqlite3 aware db program and change the folder name manually, including to a bare drive letter.

There are some features lost with this, plus you'll also be syncing the Recycle Bin and System Volume Information, which could cause other issues.

Mounted NTFS Folder

Finally, yesterday, I hit open the best possible solution short of the Amiga's Second Coming: create a separate partition and volume and instead of assigning it a drive letter mount it to an empty NTFS folder, and then subst that.

C:\Users\%username%\Dropbox     <-- mount volume here
subst b: C:\Users\%username%\Dropbox

For extra flourish, size the volume to your Dropbox limit. Doing so you always know at a glance how close you are to your quota. (Good idea to make it 15% than actual quota for breathing room. It's enough space to defrag, and Windows will get markedly slower on volumes with less free board.)

Python fixit snippets

2013-05-16T17:54:00Z

Very short problems about python, found and fixed.

>pip
failed to create process.

whups, this worked yesterday. Whatup? --> drive letters/paths have been changed. The path to python.exe specified in pip.exe has moved. Solution: reinstall pip (and any other py programs similarly impacted).

Portable Leo from scratch on Windows

2013-04-28T07:00:00Z

Here is a recipe for installing Leo from scratch on Windows (Win7 tested). The only prerequisites are command line wget in PATH and an internet connection.

About 50mb of files are downloaded and 200mb consumed when done.

Open a command prompt and run:

mkdir X:\testing
pushd X:\testing

wget -O apt.exe --no-clobber http://download.osgeo.org/osgeo4w/x86/release/apt/apt-r1193M.exe
SET OSGEO4W_ROOT=%~dp0\root
apt setup
apt update
apt install pyqt4 sip

call root\osgeo4w.bat

wget --no-check-certificate http://gist.github.com/maphew/5393935/raw/install-pip.py
python install-pip.py GO
python install-pip.py GO

pip install -i https://testpypi.python.org/pypi leo-editor

python apps\Python27\Scripts\leo

After this the directory X:\testing\root can be renamed and/or moved anywhere.

To run leo in future sessions, simply call x:\path\to\root\osgeo4w.bat python apps\Python27\Scripts\leo, either as a batch file or windows shortcut.

IMPORTANT NOTE - this is just a proof of concept. It pulls together a couple of my different projects, and is very satisfying in that regard (yay! they work!), but is not ready for prime time. Feedback welcome.

(((this is a post in progress, see this thread for more)))

Plus 1 for static site generators!

2013-04-20T10:28:00Z

There is no time like the present to really delve into and master static web generation:

Huge attack on WordPress sites could spawn never-before-seen super botnet. Ongoing attack from >90,000 computers is creating a strain on Web hosts, too

Ongoing malware attack targeting Apache hijacks 20,000 sites. Mysterious "Darkleech" exposes visitors to potent malware exploits

I received the ArsTechnica newsletter with the above stories in my mailbox in the same 5 minute window that Google Webmaster Tools informed me that one of our websites for my work has likely been compromised.

No, it's not running Wordpress, but it is running a blog-style php platform that is several versions behind current because I didn't want to break the site while trying to upgrade a system I don't understand. Now this particular bit of procrastination has turned into a bite!

Guess I know what I'm doing for the next week or more. :-/

Install Pip script

2013-04-16T21:22:00Z

Introducing a pure python means to install pip and it's requirements all in one go.

Installing pip on Windows is relatively straightforward, now, but still a tad bumpy. Basically it consists of

Download distribute or setuptools
Install it with python distribute_setup.py
Download the get-pip script
Install with python get-pip.py

(Aside: curiously the distribute docs say you can use pip to install distribute!)

On Windows this means installing wget or curl in order to follow the install instructions as given, or pointing and clicking your through click-this-url >> save as >> choose a spot >> open a python shell >> ...etc.. Easy, but boring and easy to make small mistakes which means repeating all or part of the recipe.

So as part of my continuing quest to learn enough to be called a true pythonista, I wrote a small script to do it all in one go, install-pip.py. It is adapted from Fernando Macedo's answer in How to install pip on windows?. The seminal idea is it use urllib to do the download, which is included in Python's standard library -- isn't it just great to have batteries included? (though it would begs the question of why there isn't a python package installer battery included...)

The script does a bit more, but here's the essential idea (the blog renderer is messing up the indentation, sorry):

sources = [
    'http://python-distribute.org/distribute_setup.py',
    'https://raw.github.com/pypa/pip/master/contrib/get-pip.py',
    ]

for src in sources:
    f = urlopen(src).read()
    try: exec(f)
    except: pass

Passing on an exception instead of dealing with it isn't a great idea, maybe even a stupid one, but without it only the first file is executed. Comments, fixes, and enhancements are all welcome. :)

See https://gist.github.com/maphew/5393935

Deploy website with rsync

2013-03-17T07:00:00Z

Recipe for copying my Acrylamid-generated static html website to my webhost (nearlyfreespeech.net).

In a nutshell, I use QtdSync to generate the basic rsync update script, and then add rsync advanced options as make sense to me. From QtdSync's progress screen the rsync command line can be copied and pasted to create a batch file (strip the -e ... sshpass... line for interactive password prompting).

QtdSync is a bit twitchy around the "backup/don't backup into subfolder" options. Sometimes they appear to have the opposite effect of what they say. I still found it much easier than trying to construct the appropriate rsync script from scratch myself, just test on a small data set until the settings are ironed out. The backup set .qtd files are straightforward XML so it's pretty easy to check the results of your settings.

QtdSync hard coded rsync options

These are always used.

-rltv which means:
--recursive
--links - copy symlinks as symlinks
--times - preserve modification times
--verbose

--progress - show progress

QtdSync editable advanced rsync options

--delete - delete extraneous files from destination dirs
--force - force deletion of directories even if not empty
--ignore-errors - delete even if there are I/O errors
--hard-links - preserve hard links

Extended rsync options

Add these in the Rsync Expert Settings pane. When added as Task they override the global settings.

--compress - compress file data during the transfer, saves bandwidth, speeds up transfers, 'nuff said

--skip-compress=%_ignore_ext% - skip compressing files with a suffix in LIST, not yet using, but seems like a good idea

--checksum - skip based on checksum, not mod-time & size, because preserving timestamps between Windows and 'nix doesn't always work. I suspect it's faster to use --modify-window=NUM with a window like 2secs (that's what xxcopy /ff uses) but I'm not that concerned with performance and checksum should be more reliable.

For these next two, if something goes wrong the website will stay in previous state. Not quite the same as atomic updates in a db but close enough for me.
--delay-updates - put all updated files into place at transfer's end
--delete-delay - find deletions during, delete after

--human-readable - output numbers in a human-readable format

Going Acrylic

2013-03-16T07:00:00Z

Recipe used for converting httrack snapshot of Mindtouch wiki to markdown for acrylamid

Generate the processing list

dir /s/b \www\maphew.com\*.html > process-list.xt

Edit process-list and remove junk, fix bad filenames (resultant from double quotes in name).

Copy to excel and:

convert text to table using \ for delim,
apply conditional formatting highlighting .html
sort by html column
remove duplicates
save tab delimited
search and replace tabs with \, removing dupes

Scripted Html to Markdown to Acrylamid

for /f %a in (process-list-cleaned.txt) do @mkdir .%~pa

for /f %a in (process-list-cleaned.txt) do ^
    @pandoc --to markdown --standalone --template acrylamid-pandoc-template.txt "%a" -o ".\%~pa\%~na.md"

acrylamid init converted
rd /s/q converted\content
move www\maphew.com converted
pushd converted
rename maphew.com content
rd /s/q ..\www

copy \www\acr\confy.py .\conf.py
xcopy \www\acr\theme\* theme\*

acrylamid compile --search
:: fix title collisions as needed

Clean up header & footer crud

Remove Mindtouch leftovers such as "javascript must be enabled" and "Powered by...", etc. etc. by using search and replace across all open files.

Vim regexes:

:bufdo:%s/^This application requires Javascript to be enabled\.\_.* Table of contents$//
:bufdo:%s/\*No headers\*//
:bufdo %s/Powered by \[MindTouch Core\_.*maphew)//e
:bufdo %s/---\n\n\n*/---\r\r/e
:bufdo %s/$title:\(.*\) - maphew$/title:\1/e

Sources:

http://vim.wikia.com/wiki/Search_across_multiple_lines
http://vim.wikia.com/wiki/Search_and_replace_in_multiple_buffers

Fix dates

This marks the end of the automated repeatable process.

Read through Special_RecentChanges and manually edit each .md to reflect last time touched. We don't have dates for content which was migrated to Mindtouch (prior to 17.10.2009).

Add tags

The folders are the primary tag, so search *.md and sort by location, then drag'n'drop each group into a handy text editor and paste the appropriate tag(s) in each.

I'm sure there's a way to automate this, but I decided it would be faster to brute force my way through (esp. since we've already broken with repeatable process anyway).

Spring, Cycles

2013-03-01T07:00:00Z

Well, here I am once again looking at the detritus of my website and wondering what to do with it. Each page represents, at least, an hour or three of endeavour. Many have no relevance any more and or of little if any interest to anyone, including me. Others though... well lets just say I'm not quite ready to pull the plug and let them drain away.

While not a regular experience, this is a recurring one. We've been here before. Each time I've devoted a few days or months to rewriting or transforming old content into a new container, add some new content along the way. From static html to wiki, to a cms, to another wiki, back to static html, ...to whatever this current cycle results in.

One of the historical features of this cycle I'm noting today is: after the phoenix dies and is resurrected, the life soon fades. The form remains, because in the digital world there is no decomposition, but the energy is gone. The words remain, unchanging. No or very few pages are re-introduced for a long time, months to years.

I find this very curious. Most of my waking life is spent interacting with computers. And most of that interacting is on the internet, consuming and commenting and feeding email, mailing lists, blogs, and myriad fora. How strange then, that my own piece of web real estate is so poorly maintained.

I don't know that this current renewal will, in fact, be a renewal, resulting in growth. I don't know that it won't either. I just know that this time around I'm somewhat more interested in watching the play of the impulses and energy behind the effort than the particulars of what system will be used this time around.

((which, if you're curious, will be acrylamid, though after reading about acrylamide I'm wondering about the choice in project name... ;-))

Okay, enough of the intro, on with the game!

Record internet radio with VLC, updated

2013-01-16T07:00:00Z

I needed to record a CBC radio piece today. For most cases, it appears the easiest route is to use streamtuner and streamripper, however there is something about my particular computer or CBC’s streaming style that made it difficult for me. Since it took a couple of hours to figure it all out, I figured I’d better save it for future reference. So here is a vlc only method which works, but with more clicks and it’s is easier to select a combination of options that don’t work than do. This is the combination which finally worked for me:

start VLC
File > Open Network Stream
for HTTP/HTTPS/FTP/MMS: http://www.cbc.ca/listen/streams/r1_whitehorse_32.html
check ’steam/save’, select ‘Settings’, then ‘Outputs’:
check ‘Play Locally’
check ‘File’, set filename as you wish
Encapsulation method: ‘Raw’
check ‘Audio codec’, and select ‘mpga’ from the list, bitrate as you like (96kb is okay for this particular stream)

It’s the audio codec part which tripped me up the most, even though I was saving mp3 selecting mp3 as a code yields a 0 byte file.

For command line use, copy the source and destination urls from the dialogs created above, however it is necessary to change the syntax somewhat. Namely the :sout option needs to be single quoted.

vlc http://www.cbc.ca/listen/streams/r1_whitehorse_32.htm ‘:sout=#transcode{acodec=mpga,ab=96,channels=2}:duplicate{dst=display,dst=std{access=file,mux=raw,dst="/home/matt/Radio/cbc-north.mp3"}’

Here is the same in a bash script, you should only need to edit the INSTREAM and OUTFILE lines to use elsewhere

#!/bin/sh NOW=$(date +%F_%k-%M) INSTREAM=http://www.cbc.ca/listen/streams/r1_...ehorse_32.html OUTFILE=/home/shared/Audio/Radio/cbc-north_$NOW.mp3 echo … echo …Recording CBC North internet radio to echo $OUTFILE echo …expect a 5 second delay before hearing anything echo …and a few “access_mms” errors prior to that. echo … /usr/bin/vlc $INSTREAM ‘:sout=#transcode{acodec=mpga,ab=96,channels=2}:duplicate{dst=display,dst=std{access=file,mux=raw,dst="'$OUTFILE'"}}’

2013-Jan-16\

Thanks for writing up the howto on the use of headless vlc to capture streaming audio.\ \ It took a little experimentation to make it work on my hardware (an arm-based pogoplug v4). I don't know if the changes are due to newer versions of vlc or pertain more to how it was built for the pacman database that arch linux uses (the os that's running on my pp4). Either way, I thought I'd send what I did your way in case you were interested in updating/adding to the howto.\

#!/bin/sh NOW=$(date +%F_%k-%M) INSTREAM=http://www.wwoz.org/listen/lo OUTFILE=/home/user1/wwoz_$NOW. mp3 echo echo recording WWOZ echo cvlc $INSTREAM --sout "#transcode{acodec=mpga,ab=96,channels=2}:std{access=file,mux=raw,dst="$OUTFILE"}"\

I wound up using cvlc as opposed to vlc so that it wouldn't complain about dbus access (lack of X11 really). There's something I have yet to sort on the output file side as libmp3lame complains "[libmp3lame @ 0xb3e80] channel_layout not specified". It does record, but what the exact parameters it writes the file with I haven't yet looked into.\ \ Regards,\ John\ \ \<John D. Doyle III>

Sources:

What does your Map projection say about you?

2011-11-14T07:00:00Z

Xkcd has a wonderful comic on map projections in common and not so common use, and what each of them have to say about the personalities of those who like them. A blog post and accompanying conversation at Explain XKCD fills out some of the jokes.

I liked so much I've reworked it into a 10"x12" mini-poster for printing:

Wanted: dead simple argparse examples

2011-09-15T07:00:00Z

The standard documentation for the argparse python module, while excellent I'm sure, is too much for my tiny brain to grasp. I don't need to do math on the command line or meddle with formatting lines on the screen or change option characters. All I want to do is "if arg is A, do this, if B do that, if none of the above show help and quit"

appy

2011-09-01T07:00:00Z

I’ve been sporadically working on a little command line installer for OSGeo4W, called apt. It is based on cyg-apt, a command line apt-get-like installer for cygwin byJan Nieuwenhuizen. I currently have two forks. One for o4w alone, which I’ve renamed apt, and the second for cygwin alone, which I’ve left as cyg-apt. Forking is not a good idea when there is so much in common, so I’m going to try and bring them together. One app, two installers. Neither name fits this new job so I’m going to use appy for the time being. It will have to change eventually as there’s another, much bigger, python project with the appy name, but I don’t want to spend hours agonizing over a name. Again.

The appy series of posts will chronicle my misadventures as I try and teach myself programming by making this small tool a little bit more useful. To me anyway!

Matt’s cyg-apt: my only claim to (in)famy here is converting system calls to md5sum, tar, etc. to pure python. So now the only pre-requisite is python. All the real work and intelligence in this program is a result of Jan’s work.

I created a standalone exe of the o4w version which can be used to install o4w from scratch, but at 2.5mb it’s a little big for what it actually does.

Things are going really well. I'm nothing short of astounded with what me, a nonprogrammer, has been able to accomplish in a few dozen hours (albeit spread over many evenings over a year). Why just last night in cyg-apt I added a function to use the last used mirror instead of hardcoding it. Concieved, thought about, and implemented all in one session. Now that is cool!

(many moons later) I love python, it makes me feel like I know what I'm doing. ;-) Around two hours ago I discovered apt was broken because the server changed it's ini format. One hour later I had it fixed, tested and committed with a routine I've never used before.

2011-09-01 a major milestone reached, Apt eats Apt!

Last night I completed building a package of apt.exe can upgrade itself. This means, I think, that now apt should be able to replace any and all packages in Osgeo4W in one go including python. There is much left to do, it's only been tested on two computers so far, and it consumes way more diskspace and bandwidth than I think it should, but it works!

To try it out, point your existing apt or o4w setup program (see Testing Repositoryat end of this page) at http://download.osgeo.org/osgeo4w/testing/ and upgrade away.

apt --mirror http://download.osgeo.org/osgeo4w/testing/ update
apt new
apt upgrade

After this running apt with no parameters should yield the usual help page with a version number at the top that resembles:

-={ Id: apt.py 1179 2011-09-01 12:42:24Z maphew }=-

apt [OPTION]... COMMAND [PACKAGE]...

If the first line is just [option] etc. the new apt.exe hasn't been activated. Running %osgeo4w_root%\bin\apt.bat should fix that. If not please let me know.

Mindtouch

2011-08-03T07:00:00Z

v10 license file for Core

Mindtouch now consolidates all of it's products into a single set of packages, which is good, however this necessitates contacting HQ for a license file even for the open source Core version. Some are bothered by this and don't want to give Mindtouch Corp their email address. If you are in this group feel free to use the attached license file (rename to 'license.xml'). It is "licensed" to mindtouch-core@safetymail.info.

Downgrade from Enterprise to Core

As a result of this thread I periodically get private messages asking how to downgrade to the open source. It's really very simple (though I don't know if this is still true with v10):

hi maphew, we are not allowed to talk about that in the forum... don't know why! so, just delete the license.xml, restart dekihost and you're back on core =)

best regards,
---snip---

Police make it hard to report

2011-07-06T07:00:00Z

Police make it hard to report things to them.

Last week while on a canoe trip we spotted a minivan crashed in the river, at the bottom of a 30m+ cliff. From the open driver's side door It looked like it had been pushed over the edge, and, judging from the multi-level water stains on the side, had been there for a week or two at least. Still, being good citizens when we got to somewhere we could make a call we phoned the police just in case nobody had reported it yet. They received the report easily enough but then demanded name, phone number, street address, a few other things I don't recall, and birthdate. Needing to know name and number makes sense, address is a bit more intrusive, but okay, who knows, maybe I'm a disgruntled lover trying to get someone in trouble. But birthdate? Take a hike! That's irrelevant to the issue at hand and none of their business.

This morning while walking the dog along a bush trail I found a bike that looked stolen or abandoned. Late this evening it was still there. Now I know from past experience that a bike left on it's own in the trees will very likely soon end up a kilometer or two down the trail smashed and bent, and with some of the better bits missing altogether. An unknown faceless someone would likely be very grateful to have it moved somewhere for safekeeping and reported to the authorities, meaning the police, and eventually returned safely home to them. And I'd feel pretty good for having played the good samaritan. Trouble is I don't want the police treating me like a suspect when all I'm trying to do is hand someone else a good turn.

I'll call them, eventually, an unknown faceless someone deserves it, but I have to overcome resistance to do so. Maybe next time I see something by the trail or in the water I'll just keep going. Come to think of it, maybe that's why I have the past experience in the first place of seeing unattended things transformed into junk and garbage. Others before me had decided it's just not worth the hassle; "nothing to see here, move along." That's sad.

Pale Blue Dot

2011-05-17T07:00:00Z

This article came across my desk and inspired me to make a poster. Well, it inspired me to find a poster, but I didn't locate one that resonated just the right way with me, so I made my own.

A portion of the first ever 'portrait' of the solar system taken by Voyager 1 in 1990 from a distance of more than 4 billion miles from Earth. Our world is a mere point of light, a crescent only 0.12 pixels in size, coincidentally right in the center of a scattered light ray, resulting from taking the image so close to the sun.

"We succeeded in taking that picture , and, if you look at it, you see a dot. That's here. That's home. That's us. On it, everyone you ever heard of, every human being who ever lived, lived out their lives. The aggregate of all our joys and sufferings, thousands of confident religions, ideologies and economic doctrines, every hunter and forager, every hero and coward, every creator and destroyer of civilizations, every king and peasant, every young couple in love, every hopeful child, every mother and father, every inventor and explorer, every teacher of morals, every corrupt politician, every superstar, every supreme leader, every saint and sinner in the history of our species, lived there on a mote of dust, suspended in a sunbeam.

The earth is a very small stage in a vast cosmic arena. Think of the rivers of blood spilled by all those generals and emperors so that in glory and in triumph they could become the momentary masters of a fraction of a dot. Think of the endless cruelties visited by the inhabitants of one corner of the dot on scarcely distinguishable inhabitants of some other corner of the dot. How frequent their misunderstandings, how eager they are to kill one another, how fervent their hatreds. Our posturings, our imagined self-importance, the delusion that we have some privileged position in the universe, are challenged by this point of pale light.

Our planet is a lonely speck in the great enveloping cosmic dark. In our obscurity -- in all this vastness -- there is no hint that help will come from elsewhere to save us from ourselves. It is up to us. It's been said that astronomy is a humbling, and I might add, a character-building experience. To my mind, there is perhaps no better demonstration of the folly of human conceits than this distant image of our tiny world. To me, it underscores our responsibility to deal more kindly and compassionately with one another and to preserve and cherish that pale blue dot, the only home we've ever known.

-- Carl Sagan, "Reflections on a Mote of Dust"

Listen to Carl read this passage
Watch the speech set to music and imagery*
Purchase a hardcopy in a varity of sizes, paper & framing options
Download in editable svg format (350k)

Sources & Inspiration:

iJourney - http://www.ijourney.org/index.php?tid=682
*Carl Sagan "Pale Blue Dot" (New Version) Spread This! - http://www.youtube.com/watch?v=luAteAz3WQ0
Sky Image Lab - http://www.skyimagelab.com/pale-blue-dot.html
Matthew Beckler - http://www.mbeckler.org/perspective/
Howard Abrams - http://www.howardism.org/Other/Pale_Blue_Dot.html (Howard has an audio only version; took quite awhile to find.)

Pale Blue Dot by Matt Wilkie is licensed under a Creative Commons Attribution-Share Alike 2.5 Canada License Based on a work at www.skyimagelab.com. Permissions beyond the scope of this license may be available at maphew@gmail.com.

Time Traveller Drop Points

2011-05-17T07:00:00Z

I was cleaning out old email and ran across this important message which fell through the cracks. I don't have the time to fulfill this urgent request, but thought perhaps I could help by getting the word out to someone who can. Thus was born the Time Traveller drop points:

A web map service for Time Travellers who need to arrange for pick up and or delivery of specialty goods. On the go and need a supply dump provisioned? Put your drop point and request here.\ \ NOTE: payment and delivery term arrangements are *strictly* between traveller and deliverer. We don't want to know about it and will not refund a deal gone bad.

—–Forwarded Message—–\ Subject: Dimensional warp Generator Needed ubamz\ Date: 09 Aug 2003 16:33:10 +0000

Hello,

I’m a time traveler stuck here in 2003. Upon arriving here my dimensional warp generator stopped working. I trusted a company here by the name of LLC Lasers to repair my Generation 3 52 4350A watch unit, and they fled on me. I am going to need a new DWG unit, prefereably the rechargeable AMD wrist watch model with the GRC79 induction motor, four I80200 warp stabilizers, 512GB of SRAM and the menu driven GUI with front panel XID display.

I will take whatever model you have in stock, as long as its received certification for being safe on carbon based life forms.

In terms of payment:

Payment can be made in Galactic Credits, Platinum gold, or 2003 currency upon safe delivery of unit.

INSTRUCTIONS MUST BE FOLLOWED EXACTLY:

Please transport unit in either a brown paper bag or box to below coordinates on Saturday August 9th at (exactly 3:00pm) Eastern Standard Time on the dot. A few minutes prior will be ok, but it cannot be after. If you miss this timeframe please email me. I will not be there prior to 2:45pm EST, (so do not transport before then).

\ View Time Traveller drop points in a larger map

Item is to be delivered at Majority Lane in Woburn, Massachusetts located at: Latitude N 42.49430 & Longitude W 071.14275 and the Elevation is 140 feet.

WARNING: DO NOT ATTEMPT TO TRANSPORT ITEM BY REGULAR MEANS OF TELEPORTATION. THEY ARE MONITORING AND WILL REDIRECT THE SIGNAL!!

I DO NOT CARE HOW YOU HAVE TO GET IT HERE, JUST DO IT IN A WAY THAT NO SPYING EYES WILL POSSIBLY BE ABLE TO REDIRECT or BLOCK THE TRANSFERENCE. IT IS VERY IMPORTANT THAT YOU BE ABLE TO MONITOR THE TRANSFER.

HOW ARE YOU GOING TO SEND IT SO THAT THEY CANNOT REDIRECT OR BLOCK IT??? If in doubt do not transport actual unit until your method of transfer can be confirmed as a success. You just might need to send a intergalactic courier to deliver item safely to me. If so be VERY careful at how they approach me.

It is best if you send a intergalactic courier to deliver, this way you can be certain the unit arrives ok, However If you are certain that you have the means to teleport unit in a safe manner please send a (separate) email to me at: webmaster@federalfundingprogram.com only after unit has been safely delivered with payment instructions.

Do not reply directly back to this email as it will only be bounced back to you.

Decollaring with nearblack

2011-04-28T07:00:00Z

The attached geotiff image has two collars to be removed. The outermost consists of pure black (0,0,0) while the innermost is off-white (ranges from 240 thru 255). I used gdalsetnull to change 0,0,0 to nodata, then ran `nearblack -white` on the result. I detect no difference in the output image even with a very large fuzz factor (-near 50) .

What am I doing wrong?

Answers

From Even Rouault I learned that nearblack doesn't grok nodata, it just looks at the pixel values. Consequently it never sees the white collar, except a small portion at the very top of the image.

Thankfully Luke Pinner, who's come to my aid before, has an elegant solution using a VRT intermediate file (ref):

#Change black to white
gdalbuildvrt -srcnodata 0 -vrtnodata 255 to-be-decollared.vrt
to-be-decollared.tif
#Run nearblack -white
nearblack -white -o decollared.tif to-be-decollared.vrt

ipy and arcgisscripting

2011-04-21T07:00:00Z

….some further experiments with ipythonand ArcGIS python geoprocessing. Here is a simple script to convert a bunch of coverages to shapefile. There are 9 coverages occupying 3mb, it takes about 6 minutes, consumes 2 processors to 80-90% capacity and chews through 300mb of ram:

cd w:/Env-dat.003/2007-March/workspace/envy_ed2import arcgisscripting
gp = arcgisscripting.create()
gp.Workspace = ‘./’

todo = !dir /b fwtc*

for cov in todo:
   try:
      print cov
      gp.FeatureClassToShapefile( cov + ‘/arc’, ‘./shp’)
   except:
      print gp.GetMessages()

Contrast that to using fwtools ogr2ogr, which takes 10 seconds:

for %a in (fwtc*) do ogr2ogr -f “esri shapefile” ogr\%a %a

The problem is, ogr2ogr doesn’t currently translate everything arcpy does. It leaves behind the metadata and fills voids (though these can be filtered out later as the attributes are empty, but that is an extra step). There’s also no way (that I can see) to select only arcs or only points.

(June 6th) another episode in the ogr vs arcgis battle:

Goal: Project 70 shapefiles in various subdirectories occupying 300mb.

ArcGIS BatchProject interactively:

search catalog for k:\incoming\NHN*.shp
fire up Batch Project toolbox
drag’n'drop search results (wait 2+ minutes for verification step to complete)
run
oops, error in selected projection parameters. failed.
try again, this time using Command Line Window to see what’s going on. (2+ minutes for verification)
figure out problem, fix parameters (2+ min for verification, which repeats every time you attempt to Insert Variable or other command change)
execute the command. (Finished in 21 minutes, 43 seconds.)

total arc time: >45 minutes.

ogr2ogr:

research command syntax, examine old notes and scripts (5 minutes)
build and test command line (2 minutes)
execute (7 minutes, 12 seconds)

total ogr time: \~15 minutes.So even if we subtract the debugging time for the problematic projection parameters (which have never caused me trouble before, but that is another story), ogr is still twice as fast. Which could be 3 times as fast if the command syntax were internalized and I didn’t have to look it up. Guess where I’m going to be spending some optimisation energy!

And now for the initial impetus for posting this comment, saving my command line!

for /r %a in (nhn*.shp) do @ogr2ogr ogr %a -t_srs z:\Arcutils\prj\YkAlbers.proj4

Don't show internal boundaries, or: Arcinfo Regions, we miss you!

2011-04-18T07:00:00Z

this page is to support the GIS.stackexhange.com question of the same name, namely how does one most efficiently use arcgis and feature classes to achieve some the same things which used to be possible with arcinfo workstation and region sub-coverages.

Please see attached archives for a v10 map package which demonstrates the problem and (simulated) desired results, and the source arcinfo coverage from which the simulation is built.

COMfounded python

2011-01-25T07:00:00Z

I think my python-COM environment is messed up.

Got

D:\> python list-fc.py
../Scratch/blank_canvec.gdb
<COMObject <unknown>>

and expectedsomething like this:

../Scratch/blank_canvec.gdb\ [u'counties.shp', u'roads.shp']\ counties.shp\ roads.shp

Possible culprits:

using Mark Cederholm's python+arcobjects recipe requires registering com stuff, which conflicts wth PythonWin's com stuff.
I'm using Arcgis with python 2.6 instead of the shipped 2.5. This hasn't proved problematic before, but maybe I just haven't wondered into this section of the forest before (which means I need to eat crow for Jason Scheirer).
I have concurrent python 2.5 and python 2.6 installs, 32 and 64bit, and they are walking over each other
one of the many other programs which distribute static python dll's is in path and colliding
I'm just very poor snake charmer. :)

Troubleshooting

Win32 bug#3076741, "This object does not support enumeration". Using win32com.client.gencache registers com objects on the fly, fixes one problem and creates others.

update, 2011 Jan 25

I think I've narrowed it to the win32 extension, as presented by the arcgisscriptingwrapper.

I uninstalled all versions of python and cleared all directories from PATH that had any kind of python??.dll, then installed python 2.5.4, and numpy from the arcgis dvd (as per instructions here). If PYTHONPATH contains ...\ArcGIS\bin, then we get the expected results, if not, and we add arcgisscripting.py to Lib\site-packages: win32com not found. If we then install the win32com package we get the \<COMObject \<unknown>> error.

The readme for PythonWin on the install dvd says "ArcGIS no longer installs PythonWin because the recommended methodology for creating the geoprocessor has changed to using a new Python module that ESRI has developed ...".

Sooooo, arcgisscripting.py and win32com work for most things, but not for gp.ListFields() or gp.ListFeatureClasses() functions. I bet this also applies to other list functions too, see changes in list methods knowledgebase article. This means that python 2.6 and 2.7 can be used with ArcGIS 9.x, but only if you stay away from the list functions.

Retrieve members of AD group

2011-01-18T07:00:00Z

How to retrieve members and email of an Active Directory group in a re-usable format.

Locate dsget command line utility and put it in path.

Open an administrative command shell and:

dsget group "CN=mygroup,OU=mydepartment,OU=Users and Groups,DC=mydomain,DC=ca" -members |dsget user -samid -display -email

Searching Active Directory

2011-01-18T07:00:00Z

This is from the I can't believe I didn't know that and whyhaven't I been using it for the last ten years!?? department.

Many times in the last decade I'veneeded to*search Active Directory*for a person's name, login name, what members belong to what group and so on. Each time the need has been pressing enough I've gone out and searched the 'net for solutions, and each time after a time come up with a solution. Said time might have been 10 mintues or 2 hours depending on what in particular I was trying to do. The solutions I've found and used ranged from the easy to use AD-Explorer to roll your own python or batchfile scripts (that have taken further hours to develop to suit local circumstances).

Encountering the need again today, and realizing my local stash of recipes was much too scary or technical to hand off to our regular staff who would actually be using it, I embarked on the quest again. Lo and behold, every version of Windows from (at least) XP on up includes the built in Find Users, Contacts, and Groups. It's buried under 'My Network Neighbourhood > Common Tasks' which I'd given up on long ago as being a useless "my computer" place to visit (when I want to see the network it's far more common to want to see the workstations and servers near me and turned on than the last 30 or more places I've been to). Who knew? (obviously not me!)

Anyway, here's the meat of it and you can't put it anywhere you want, so it's actually convenient: Create a shortcut to rundll32.exe dsquery, OpenQueryWindow. And that's it, you're done.

Sheesh.

Thank you Information Systems & Technology at University of Waterloo for illuminating me.

Long Live the Aurora Text Editor

2010-10-20T07:00:00Z

Thanks to the persistence of Kent Nassen and the generosity of Jeff Wunderlich, the Aurora Text Editor can be downloaded and registered, “…the .bat file command I use to register is: ac -reg “Jeff Wunderlich” “xujcwpkgzngqmqn” 1 Feel free to share it with anyone on the net.”

So what’s the big deal? Well I covered most of it a few years ago in Musing on the Favourite Text Editor (slow link, involves time travel), but at the moment my biggest cause for joy is that I can, once again, select a column and inserting a character or range of characters (useful for turning fixed width text records into delimited text), and filling a columnar selection with an increasing pattern (1,2,3… a,b,c…) similar to Excel’s extend series command.

Since I wrote the musing’s essay I’ve been a happy Cream user,with occasional forays into Notepad++ when speed and portability (happy on a usb key) is desired. Now that Aurora is around and permanently free I’m positively cheery. Yes I know TextPad can do column fills, and TP is a very fine editor indeed. I bought it for work after all, but it’s not cross platform and I don’t want to pay for it on for every computer I use. Besides, Aurora is only a 650kb download. Where else can you get so much editor bang for your buck? Seriously, where else?