Ned's post about "human sorting" lists lead to the inevitable comments from people who want to write the most succinct solution to the problem. Among them was the remark that "exception handling is slow." Ned asked me why it was assumed that exception handling was so slow, and I replied that it isn't (based on some of the remarks about improvements in exception handling in Python 2.5). I had no real evidence of the overhead from catching exceptions, so I thought I'd benchmark the different sorting methods
First, Ned's:
def tryint(s):
try:
return int(s)
except:
return s
def ak1(s):
return [ tryint(c) for c in re.split('([0-9]+)', s) ]
Then, using str.isdigit instead:
def safeint(s):
if s.isdigit():
return int(s)
else:
return s
def ak2(s):
return [ safeint(c) for c in re.split('([0-9]+)', s) ]
Somebody mentioned the new syntax in 2.5 that allows one-liner conditionals, thus saving a function call:
def ak3(s):
return [ int(c) if c.isdigit() else c for c in re.split('([0-9]+)', s) ]
Of course, there's the easy optimization of compiling the regular expression beforehand:
NUM_RE = re.compile('([0-9]+)')
def ak4(s):
return [ int(c) if c.isdigit() else c for c in NUM_RE.split(s) ]
Ok, so what were the results? Here, I'm sorting a 5000 item list of strings created like so:
def make_list(sise):
samp = string.letters + string.digits
return [''.join(random.sample(samp, 4)) for i in xrange(size)]
It turns out the conventional wisdom isn't just an old pythonic wive's tale: exception handling is slow. Even though ak1 would've been my first thought for the function
(hanging around with Ned too much?) because it seems more pythonic to just try to convert the value to an int and catch the exception if it's not an integer, that function is 4 times slower
than the fastest case. Also, this benchmark shows that there isn't much of a gain from the new conditional syntax, even though a function call is saved.
As usual, pre-compiling the regex makes lots of sense for performance-critical code. For giggles, I have a function (that I suppose works...) which avoids the regular expression:
def ak5(s):
new_s = []
b = ''
for c in s:
oc = ord(c)
if oc > 57 or oc < 48:
if b:
new_s.append(int(b))
b = ''
new_s.append(c)
else:
b += c
return new_s
It's not as slow as I thought, averaging 25.5 msecs on my machine.
What have we learned today?
Python exception handling isn't nearly as lightweight as I thought. Especially avoid it in tight loops.
Function calls should be minimized. (duh)
Sometimes the more succinct solution isn't faster.
This was playing as Katie drove. I can't get enough of it.
Home is where I want to be
Pick me up and turn me round
I feel numb - burn with a weak heart
(So I) guess I must be having fun
The less we say about it the better
Make it up as we go along
Feet on the ground
Head in the sky
It's ok I know nothing's wrong . . nothing
Hi yo I got plenty of time
Hi yo you got light in your eyes
And you're standing here beside me
I love the passing of time
Never for money
Always for love
Cover up + say goodnight . . . say goodnight
Home - is where I want to be
But I guess I'm already there
I come home - -she lifted up her wings
Guess that this must be the place
I can't tell one from another
Did I find you, or you find me?
There was a time Before we were born
If someone asks, this where I'll be . . . where I'll be
Hi yo We drift in and out
Hi yo sing into my mouth
Out of all those kinds of people
You got a face with a view
I'm just an animal looking for a home
Share the same space for a minute or two
And you love me till my heart stops
Love me till I'm dead
Eyes that light up, eyes look through you
Cover up the blank spots
Hit me on the head Ah ooh
From: xxxxxxx@xxxx.com
Date: Aug 7, 2007 6:50 AM
Subject: Fwd: Fw: Disturbing Information?????????????
lets vote thm out
they take starting jobs
and are getting our ssi benefits
if they want to benefit from citizen ship
then let them become americans like
u and
every immagrant that came to america
learned english
my daughter volunteers her time
to teach english to immagrants now
this is america the offcial language
should be american (english)
xxxxxx xxxxxx
Note: forwarded message attached.
---------- Forwarded message ----------
From: xxxx@xxxxx
To: xxxx@xxxxx
Date: Tue, 7 Aug 2007 05:15:00 -0500
Subject: Fw: Disturbing Information?????????????
Shocking Senatorial Votes
The following senators voted against making English
the official language of America :
.... edited: look these up yourself ....
Next, following are the senators who voted
to give illegal aliens Social Security benefits.
They are grouped by home state.
If a state is not listed, there was no voting representative.
.... edited: look these up yourself ....
SEND THIS TO EVERYONE YOU KNOW.
THE ENTIRE POPULATION OF THE UNITED STATES
NEEDS TO KNOW THIS INFORMATION....
UNLESS YOU DON'T MIND SHARING YOUR
SOCIAL SECURITY WITH FOREIGN WORKERS
who haven't paid in a single dime !! Please circulate this!!!
What are the qualities of a successful progressive movement or leader? Last week I sent this video of a Henry Rollins rant to Albert, the resident conservative at work. He asked if Rollins was the "liberal Michael Savage," and I think that perhaps his style is similar. And yet, I don't think he's worth listening to, even if I do often agree with what he says.
I started thinking about why Rollins doesn't resonate with me, which lead me to think about the leaders who do inspire me. I came up with a list of qualities that a progressive leader should encourage in us.
Informed: A population uninformed is a population doomed to manipulation. What are the problems in the world that we are working to change? What issues in my own community are important? What are the facts about the situation, and what is the larger context? A leader who can present the issues in a clear and rational manner lays the foundation of a successful movement. People who understand the issues hold more ammunition in a debate than those whose understanding comes from faith-based opinion.
Empathetic: Empathy puts us in the shoes of people we otherwise wouldn't care about. Rather than appealing to the individual concerns ("What does this have to do with me? What do I gain from this issue?"), a leader can direct people's energy into causes that benefit others. Why should I care about torture? I'm not in any danger of being tortured. A lazy leader might say, "You should care because soon you might be the target of government torture!" — an appeal to fear is the wrong approach. An empathetic response might be, "You should care because we live in a country of laws, and no one is above the law. Our Constitution holds any treaty that the U.S. enters into as equivalent to the highest law in the land, yet the government defies the Geneva Convention and our own Constitution by condoning torture. The government should be working for the people, not breaking it's own hard-fought laws." Sure, that kind of response takes more words and is a little more nuanced than simple fear of government control, but empathy is a tough thing to inspire. When our society of consumption entices us to revel in our basest desires and focus on pleasing the self above all else, it can be an uphill battle to convince people to think outside themselves. But it's happened before, and it'll happen again.
Hopeful: Without hope, informed, empathetic people mire in a low-level angry cynicism. Absent hope, apathy grows, leading to a less-informed populace with no power or motivation to change anything. A decent leader can inspire hope by pointing to examples of successful change in the past, by appealing to the possibility of a better future, and, most importantly, leading by example. People lose hope in their leaders because the leaders betray confidence through corruption and ineffectiveness. Being incorruptible and striving for honesty go a long way towards maintaining hope and motivation in a progressive cause. The prospect that a leader can lead a life above reproach would fuel a movement; people will gladly follow someone who leads by example. A progressive movement with a progressive leader who inspires hope, empathy, and knowledge would be on the right track toward change.
Angry: Finally, a little anger can induce people to get off their asses and organize for change. Liberals (well, at least Democrats) often look down on anger as a quality in a leader — they see it as something embarrassing or "unelectable." The truth is that the "ineffectual liberal" stereotype has a basis in reality. Too often Democrats let themselves become bulldozed into irrelevance out of deference to opposing views (however wrong those views may be). Progressive leaders shouldn't be afraid to stand up for themselves against the tide of fundamentalism and intolerance. Speak loudly, passionately, and rationally, and others will feel empowered to do the same in their own communities. A progressive movement containing healthy doses of the other ingredients combined with an ounce of anger might, just might, gain enough traction to foster real change — change that benefits citizens while keeping government corruption in check.
Michael Savage and others of his right-wing ilk focus on fear and the individual desires in order to manipulate people. They frame issues always as a conflict of us vs. them: Terrorists are trying to get you; Liberals want to tell you what to do; Immigrants are going to take your job. They have no interest in informing people with facts or reason, and they have no interest in speaking up for those whose voices cannot be heard. Those pundits are intolerant, hateful, and add no value to society.
Rollins falls flat because his angry rants, though impassioned, do not inspire hope or inform me of anything I didn't already know. The rant is directed through the individual against the government. There is often no appeal to empathy for others. It's me against the evil government and conservatives. Although I wouldn't go out of my way to tune into his show, he constitutes that small dose of anger that I think might be helpful to motivate progressives, as long as he doesn't advocate blind violence or anarchy.
Bill Moyers, however, does inspire me. As a journalist and effective documentarian, he's great at informing people about progressive issues through facts and debate, and he highlights the plight of others in a way I can relate to. Being a former director of the Peace Corps and part of the government means that he knows the potential for change exists, and I find his lack of cynicism refreshing.
The challenge now to progressives in America is to seek more examples of honest, intelligent, passionate liberals who can inspire others to change the policies set by the current administration and head the country in a direction where the government works to help all the people, not just the richest corporations and lobbyists. At 73 years old, Moyers won't be at it forever. Who in my generation will stand up and make a difference?
