[ About | Features | Music | Pictures | Software | Writing ]
Crummy: The Site
We fight 'em until we can't.


News You Can Bruise

[Archives]
Buy my books!

2 years ago: Month of Kickstarter #22: Rolling the Dice

10 years ago: Blue Cheese Pesto

11 years ago: Live the Parody!

12 years ago: It's well known that the powers of magic are enhanced...

[Comments] (2) The Average Minecraft Skin: Currently my two spare-time hobbies are 1) Situation Normal revisions and 2) gathering Minecraft data. Yes, I'm still at it! There's a lot more data than I anticipated! I'm up to about 175,000 maps, and I've branched out into archiving mods and texture packs. There's even more I could do, but pretty soon I'm going to have to put away the data-gathering part of this project for six months or a year so I can get other stuff done.

My reach keeps expanding because whenever I decide that a certain dataset isn't interesting and I won't bother with it, I immediately come up with something really cool to do with the dataset. For instance, Minecraft skins, the little images that are bitmapped onto your character in the game to make you look like a penguin or Jean-Luc Picard. I never really cared much about skins, but in the process of deciding not to bother with them, I discovered that Planet Minecraft, one of the biggest repositories of skins, lets someone who uploads a skin specify a gender ("male", "female", "interchangeable", and "other"), as well as a category classification ("animal", "cartoon", "famous person", etc.). Now I was interested! Skins are data about how people present themselves in the virtual world, data that I could gather and graph.

Here's a simple graph showing the skins available on Planet Minecraft, broken down by category and gender:

Self-reported gender of Minecraft skins

In every category male skins are drastically overrepresented, but the discrepancy is smallest in "Other". Why? My guess is that "Other" is where you'd put a skin that you made to represent yourself.

Since there are only two different sizes for skin images, you can average a number of skins together to get a new skin. Here's a skin that is the average of 100 of the most popular "female" skins on Planet Minecraft:

And here's the average of 100 of the most popular "male" skins:

That's a pretty preliminary result, but I think it's interesting. The major sexual dimorphism among Minecraft skins—the shape of the eyes—comes through loud and clear. If you want to use one of these as your actual Minecraft skin, I recommend going in with an image editor and erasing the upper-right part of the image. Otherwise your character's head will be shrouded in a ghostly hat, and it won't look good.

[No comments] June Film Roundup: It doesn't get better than this. I liked every single movie I saw this month. Two, maybe three of them are in my top ten. I guess that's what happens when you only see time-honored classics and movies you've already seen and loved. I'm posting this a little early because I'm going on vacation next week. Have fun!

May Film Roundup: Ready for "Wacky Wednesdays" here at News You Can Bruise? Here's the deal. We got five movies in the May roundup, but only three of them I actually saw in May! One is from April and one I saw yesterday. Also, it's not Wednesday.

@MinecraftSigns, And Minecraft Maps: I finished a draft of Situation Normal and sent it in to writing group, so I've now got time to reveal the other non-NYPL project that's been taking up all of my time. Ta-da! It's a bot! @MinecraftSigns posts signs that I found in Minecraft maps using the pymclevel library I learned for the Historical Minecraft project.

For a long time, signs were the only form of textual self-expression possible in Minecraft. You get four lines of 15 characters each. In normal play they're generally used as labels or signposts. Custom mapmakers also use them for instructions to the player, dialogue, narration, and hidden messages. They are a medium of communication with more severe character restrictions than Twitter, which makes them a great subject for a Twitter bot. Signs posted so far range from the profound:

This one's
about dropping

To something I think I saw on one of those trendy t-shirts recently:

peanuts and
pickles and
potatoes and
Paul

To the crowd favorite so far:

Do not
Extinguish fire
You will lose.

Oh goodie, you say; another bot from Leonard! What will he come up with next? Yet another bot? The answer is yes. But, before you dismiss @MinecraftSigns as just another window into a beautiful realm of found poetry, ask yourself this: how did I get this data in the first place? Where did all these Minecraft signs come from? Oh, I don't know, maybe from the sixty-five thousand Minecraft maps I've got on my hard drive?

That's right. After the Historical Minecraft project I thought back to late 2011 when I was enjoying the world of custom Minecraft maps. I then thought forward to early 2012, when I was kind of done with custom Minecraft maps, but when I moved all the ZIP files I'd downloaded onto a backup drive rather than deleting them, because these things don't stay on the Internet forever and it would be nice to have a copy, say, twenty years from now. And then, in early 2014, two years into that twenty, I was thinking about that little act of preservation and it hit me: who's archiving the rest of those maps?

The answer was: apparently nobody. And then the answer quickly became: I am. From the middle of April to the middle of May I archived 65,000 maps linked to from the Minecraft maps forum. That's out of about 100,000 maps total. I verified that 25,000 maps are gone, and there are about 10,000 maps I didn't get because they're scattered across a million different file-sharing sites.

So, at least a quarter of the maps put up since 2010 are already gone. I was able to get screenshots for a lot of the missing maps, so it's not a total loss, but that's still really bad, and not only because it's generally bad when interesting things leave the Internet.

Minecraft is the medium used by a lot of accomplished designers and artists. The most obvious examples IMO are Vechs (Super Hostile) and three_two (Vinyl Fantasy). Those two are pretty legendary and their maps are in no danger of being lost, but there's a lot of really great stuff published in 2011-2012 that was lost in the flood. 2011-2012 was the silent-film era of Minecraft custom maps, when the genres were being defined and the first wild experiments were happening, but when the medium was not taken seriously enough to warrant systematic preservation. In the future we'll have tools for finding the overlooked gems, but first those maps have to make it to the future.

Speaking of the future, Minecraft is the training ground for the next generation of game designers, the way ZZT was the training ground for my generation. There's a ZZT archive; it's got about 2,000 ZZT games. How many are lost? Sure would have been nice to save more of them, but all we had back then was BBSes. We didn't have a big official "ZZT forum" with a special place for posting links to your games.

Finally, even a map that's made by a young child who grows up to be an actuary rather than a game designer is valuable. For one, it's valuable to the actuary. I didn't grow up to be a visual artist, but I value this awful, mysterious poster I drew when I was six. That poster would be long gone if someone (my mother) hadn't archived it for me. Second, these maps might be useful in the aggregate as a source of information about period slang or the way children visualize three-dimensional space. Third...

Well, I think one reason Minecraft is so popular with kids is it recreates an experience that American kids generally aren't allowed to have anymore: going outside and playing in a semi-natural environment, on your own or with friends, without parental supervision. There's this infamously bad Minecraft map from 2011 called Quest for Gallell, which turned out to be made by a six-year-old. Presumably this goofy swashbuckling playthrough was made before the players knew they were making fun of a six-year-old's map, but if you watch the video you'll notice that the players understand how to approach the map: like kids playing together in the woods. They're acting out kids acting out adults.

Quest for Gallell is the three-dimensional record of an imaginative play session, which you can play through yourself if you want. It sucks that kids can't play outside anymore, but at least we have some records of what they do instead. Those records are worth saving.

Crosspost: Apparently I have a new weblog! It's my NYPL staff weblog and I've put up a post about a project I worked on with Paul Beaudoin on like my second day at NYPL Labs. We turned a historical contour map into a Minecraft world. This is cool on its own, but it also means I now know how to programmatically generate Minecraft maps with Python scripts. The possibilities are endless, and you'll be seeing more of them later. Like, when I'm done with this novel.

If you must get all your Minecraft news in video form, you're surprisingly picky but you're also in luck. I took Nashville's own Joe Hills on a tour of 1860 Manhattan, and he recorded the whole thing. My only regret is that I didn't prime the buried TNT he discovers near the end of the video.

[Comments] (1) April Film Roundup: Running late this month because of work on Situation Normal. But I'm sick of writing that tonight, so let's crank out some great reviews of (mostly) great movies.

[Comments] (1) March Film Roundup: April Fools! As part of an elaborate prank spanning over a year I have slowly turned NYCB into mostly a film review blog! Hahahahaha... ah...

Anyway, I'm trying out a new strategy for spending less time writing these film roundups. Instead of trying to analyze each movie in detail I'm going to write only as much about a movie as I feel like writing in the moment. Sometimes this will still be a lot, but most of the time I think a paragraph's worth of text will suffice.

Read My Lips: Two New Bots: I've been trying to finish as much of Situation Normal as possible before my job at the library starts (uh... I think this is the first time I've mentioned my NYPL job on NYCB, but I'll be writing about it later). But I have created two new autonomous agents to engage and confound you.

The first is Euphemism Bot, inspired by the fact that most of the output of Adam's Egress Methods sounds like weird euphemisms for masturbation. Euphemism Bot elevates the tone by putting out weird euphemisms for all sorts of dirty, shameful things. You'll never be understood again! It's been up for about a month, and it's already subverted its programming.

From the naughty to the nautical, there's also Boat Names, which I "launched" today. It periodically sends out names that one, and only one, person decided to give their boat. The data comes from the Queneau-sounding ten thousand boat names, which I first learned of from the trivia podcast Good Job, Brain! (I'm linking to their Twitter page because their main webpage currently shows some base64-encoded text that isn't even a puzzle.) I had this idea kicking around in my head until yesterday's lunch with Andrea Phillips, when the topic turned to weird random datasets we'd collected. And now... a bot is born.

Boat Names also has an Egress Methods connection. I found the list of given names Adam uses for Egress Methods and used it to filter out boats that are named after people. This avoids the boredom of "Eleanor", which just proves that not many boat owners have wives named Eleanor.

February Film Roundup: Three films this month, none of them great, but all of them worth your time.

Mahna Mahna: My new bot, Mahna Mahna (@mahna____mahna), reenacts the Muppet Show's "Mahna Mahna" skit over the course of a day. It might be my saddest bot.

My secret is that I created this bot hoping that someone else would eventually create a Snowth bot to enact the other half of the skit. I quickly learned that there is already a Snowth bot, but it only talks to @mahna____mahna once a day. So... well, I already revealed one secret in this paragraph, I shouldn't reveal another.

Constellation Games Bonus Story Ebooks: Thanks to requests by Ron Hale-Evans and others at Foolscap, I've compiled the four Constellation Games bonus stories into a single ebook. You can get an EPUB that looks okay and a MOBI that's kinda ugly. If you want to do a better job of formatting, then a) be my guest, and b) let me know and I'll send you the original source files, which should save you some work over downloading everything and putting it together yourself.

Writing Aliens: I've put online the slides and prepared text of my Foolscap talk, "Writing Aliens", or, "Duchamp, Markov, Queneau: A Mostly Delightful Quilt". On one level it's a simple introduction to algorithmic creativity, but it's also about creativity in general, the anthropomorphization of software, and why the features that make Twitter so aggravating for humans make it such a great platform for bots. Bonuses include a recap of Brian Hayes's article on Markov and a telling of the @Horse_ebooks saga as a reverse alien invasion. The ebooks installation

The two site-specific installations that I hinted at earlier were custom scripts displaying variants on Ebooks Brillhantes and Hapax Hegemon. The text corpus comes from a scrape of everything linked to from Free Speculative Fiction Online. The software is a heavily modified version of Bruce, modified a) to stream data from a flat text file and create the slides on the fly, instead of trying to load 20,000 slides into memory at once; and b) when restarted after a crash/shutdown, to skip the appropriate number of slides and pick up where it would be if it had been running continually.

Unfortunately I never got a picture of both displays running side-by-side; if you have such a picture, I'd really appreciate it if you could send it to me.

Just after I set up the ebooks display, I met Greg Bear, who was at Foolscap running a writing workshop. We walked over to the screen and I explained the project to him. He said "I'd better not be in there." AT THAT MOMENT the screen was showing the quote "We zoomed down eleven" from this free sample of Blood Music. It was pretty awkward.

[Comments] (1) January Film Roundup: The cycle begins anew... OR DOES IT? Check out all the films I saw in January!

Yeah, only one film! Because I was travelling all month. I couldn't even count Future Love Drug, a short film made by my fellow Foolscap GoH Brooks Peck, because I came in late and only saw the last minute of the film.

I don't know if the film roundups will continue in 2014. On the one hand, I'm going to try to see, or at least review, fewer films in 2014 so I can do more reading. On the other hand, I love taking fiction apart to see how it works, and reviewing books the way I've been reviewing movies is a good way to make professional enemies. Whereas nobody cares what I say about film. So who knows?

[Comments] (1) The Crummy.com Review of Things 2013: I've been travelling for most of the month, but I managed to scrape together a year-in-review post. Here's 2012. I'm a little disappointed right now, because I just woke up from a dream in which I'd savvily combined several middle-tier Kickstarter rewards into being able to go to the International Space Station whenever I wanted, so let's start with a self-aggrandizing montage of my waking accomplishments in 2013:

Now let's take a brief look at contributions from the not-me community:

Literature: The category that suffered the most from 2013's focus on film. I didn't read that much, and my writing is slowing down because of it. This is a strange alchemy that I can't explain but I'm pretty sure other writers recognize it. Anyway, I've got some new books I'm excited about so I'll get back on this in 2014.

For 2013 I'll give the nod to Marty Goldberg and Curt Vendel's Atari Inc.: Business is Fun, a book that... well... this review is pretty accurate, but the book has a lot of good technical and business information, plus many unverifiable anecdotes. It seems I read nothing in 2013 that I can wholeheartedly recommend without reservation... except Tina Fey's Bossypants, I guess... yes! In a late-paragraph update, Bossypants has taken the award! Wait, what's this? In a shocking upset, the ant has taken it from Bossypants! Yes, the ant is back, and out for blood!

Games: 2013 was the year I finally learned the mechanical skill of shuffling cards. Maybe this doesn't seem like a big deal to you, but I've been trying to figure this out for most of my life.

The crummy.com Board Game of the Year is "Snake Oil", a game about fulfilling user stories with lies and shoddy products. The Video Game of the Year? Man, I dunno. I'm playing computer games a little more than in 2013, but still not that many. "Starbound" is really cool, and is probably the closest I'll get to being able to play "Terraria" on Linux.

Audio: As I mentioned, I'm travelling, and away from the big XML file that contains my podcast subscriptions, so I'll fill this in later, but there's not a lot new here. But I can tell you the Crummy.com Podcast of the Year: Mike "History of Rome" Duncan's new podcast, Revolutions. The first season, covering the English Revolution, just wrapped up, so it's a good time to get into the podcast.

Hat tip to Jackie Kashian's The Dork Forest. Probably not going to have to update this one, actually.

Film: Ah, here's the big one. As I mentioned earlier, I saw 85 feature films in 2013. By amount of money I spent, the best film of the year was Gravity, which I dropped about $40 on. But by any other criteria, it wasn't even close! Well, it was close enough to get Gravity onto my top twelve, which I present now. I consider all of these absolute must-watches.

  1. The General (1926)
  2. Nashville (1975)
  3. Ishtar (1987)
  4. Ball of Fire (1941)
  5. Calculated Movements (1985)
  6. The World's End (2013)
  7. No No Nooky TV (1987)
  8. Gravity (2013)
  9. The Godfather (1972)
  10. Cotton Comes to Harlem (1970)
  11. Gentlemen Prefer Blondes (1953)
  12. No (2012)

As you can tell, only films I saw for the first time in 2013 are eligible; we call this the "The Big Lebowski rule".

There was no movie that really changed my aesthetic sense this year, the way Celine and Julie go Boating did last year, but Nashville gave me insight into managing a large ensemble cast. Hat tip to Fahrenheit 451 for getting me to understand why I keep lining up for French New Wave films even though they keep pulling the football away from me.

I still don't feel like I know that much about film. I treat films like they're books. I'm not that interested in what people do with the cameras. I have no idea what the names of actors are. I find the prospect of making a film quite tedious. They're fun to watch though.

For the record, here's my must-see list from 2012, which I didn't spell out last time:

  1. Celine and Julie Go Boating (1974)
  2. Brazil (1985)
  3. A New Leaf (1971)
  4. All About Eve (1950)
  5. The Whole Town's Talking (1953)
  6. Shadow of a Doubt (1943)
  7. Paper Moon (1973)
  8. Marathon Man (1976)

Okay, I think that's enough. Nobody reads these things until the centennial anyway.

One week to Foolscap!: In a week I'm a guest of honor at the Foolscap convention in Redmond, WA. It's got a bit of an unconference feel, so apart from the basics--board game night, a talk by me that I have to prepare--we can form fluid overlays and schedule whatever we want.

Also featured at the con will be (I think I've mentioned this before) two continuous SF/F text installations I've created to astound you. This exhibit WILL NOT BE REPEATED, unless someone asks for it at another con. So if you're in the Seattle area, sign up or just show up the day of, and you'll get to hang out with me, and the other honored guest, museum curator/SyFy monster movie screenwriter Brooks Peck.

[Comments] (1) The Bots of 2014: I took an oath of non-bot-making for most of December, but now I'm back in the game. At the end of January I'm a guest of honor at Seattle's Foolscap convention, and I've got a couple site-specific installation projects that will hopefully entertain congoers to the exclusion of all other activities.

But for now, I have two new bots to entertain you, the general public. The Hapax Hegemon (@HapaxHegemon) posts words that occur only once in the Project Gutenberg corpus I've been getting so much mileage out of. So far it's emitted such gems as "zoy", "stupidlike", and "beer-swipers". And like so many of my recent bots, it won't stop until we're all dead.

My second new bot is the Serial Enterpreneur (@ItCantFail), which posts inventions. It's basically playing Snake Oil (spoiler: Crummy.com 2013 Board Game of the Year) with a much larger corpus, derived from the Corpus of Historical American English and the Scribblenauts word list.

So far my favorite @ItCantFail inventions are the delicious Fox Syrup, the liberal-friendly Left Drone, and the self-explanatory Riot College. Write in with your own wacky inventions! I won't use them, because that's not how this bot works, but it seems like a fun way to kill some time.

More bots are on the way! But not for a while, because I gotta do novel work and get the Foolscap-exclusive bots in shape.

December Film Roundup: Counting it all up, it looks like I saw 85 feature films in 2013, plus some beefy television and a ton of shorts. Unfortunately the retrospective of 1913 silent film (semi-promised at 2012's 1912 retrospective) did not materialize. Oh darn!

I'll tackle the "best of" topic in a general 2013 wrap-up later on. For now, here's a look at December's cinematic adventures:

I'm planning on seeing a lot of movies in 2014, but I don't know if I'm going to write these detailed reviews of each one. It takes a long time to get my thoughts in order and write it down, and, as you'll see when I write the year-end roundup, it really eats into the time I spend enjoying other media. So until next time, I'll see you at the movies! (If you are Sumana, Hal, or Babs.)

[Comments] (3) Markov vs. Queneau: Sentence Assembly Smackdown: I mentioned earlier that when assembling strings of words, Markov chains do a better job than Queneau assembly. In this post I'd like to a) give the devil his due by showing what I mean, and b) qualify what I mean by "better job".

Markov wins when the structure is complex

I got the original idea for this post when generating the fake ads for @pony_strategies. My corpus is the titles of about 50,000 spammy-sounding ebooks, and this was the first time I did a head-to-head Markov/Queneau comparison. Here are ten of Markov's entries, using the Markov chain implementation I ended up adding to olipy:

  1. At Gas Pump!
  2. The Guy's Guide To The Atkins Diet
  3. Home Internet Business In The World.
  4. 101 Ways to Sharpen Your Memory
  5. SEO Relationship Building for Beginners
  6. Gary Secrets - Project Management Made Easy!
  7. Weight Success
  8. How get HER - Even If It's Just Money, So Easy and Effective Treatment Options
  9. Sams Yourself
  10. Define, With, Defeat! How To Get Traffic To Your Health

The Markov entries can get a little wacky ("Define, With, Defeat!"), which is good. But about half could be real titles without seeming weird at all, which is also good.

By contrast, here are ten of Queneau's entries:

  1. Adsense I Collection Profits: The bottom Guide Income!
  2. Reliable Your Earning Estate Develop Home And to life Fly Using Don't Your Partnership to Death
  3. Help the Your Causes, Successfully Business Vegetarian
  4. Connect New New Cooking
  5. 1 Tips, Me Life Starting to Simple Ultimate On Wills How Years Online With Living
  6. How Practice Health Best w/ Beauty
  7. Amazing Future & Codes Astrology to Definitive Green Carbs, Children Methods JV Engine Dollars And Effective Beginning Minutes NEW!
  8. I and - Gems Secrets Making Life Today!
  9. Succeeding For Inspiring Life
  10. Fast Survival Baby (Health Loss) Really How other of Look Symptoms, Your Business Encouragement: drive Health to Get with Easy Guide

At their very best ("Suceeding For Inspiring Life, "How Practice Health Best w/ Beauty"), these read like the work of a non-native English speaker. But most of them are way out there. They make no sense at all or they sound like a space alien wrote them to deal with space alien concerns. Sometimes this is what you want in your generated text! But usually not.

A Queneau assembler assumes that every string in its corpus has different tokens that follow an identical grammar. This isn't really true for spammy ebook titles, and it certainly isn't true for English sentences in general. A sentence is made up of words, sure, but there's nothing special about the fourth word in a sentence, the way there is about the fourth line of a limerick.

A Markov chain assumes nothing about higher-level grammar. Instead, it assumes that surprises are rare, that the last few tokens are a good predictor of the next token. This is true for English sentences, and it's especially true for spammy ebook titles.

Markov chains don't need to bother with the overall structure of a sentence. They focus on the transitions between words, which can be modelled probabilistically. (And the good ones do treat the first and last tokens specially.)

Markov wins when the corpus is large, Queneau when the corpus is tiny

Consider what happens to the two algorithms as the corpus grows in size. Markov chains get more believable, because the second word in a title is almost always a word commonly associated with the first word in the title. Queneau assemblies get wackier, because the second word in a title can be anything that was the second word in any title.

I have a corpus of 50,000 spammy titles. What if I chose a random sample of ten titles, and used those ten titles to construct a new title via Queneau assembly? This would make it more likely that the title's structure would hint at the structure of one or two of the source titles.

This is what I did in Board Game Dadaist, one of my first Queneau experiments. I pick a small number of board games and generate everything from that limited subset, increasing the odds that the result will make some kind of twisted sense.

If you run a Markov chain on a very small corpus, you'll probably just reproduce one of your input strings. But Queneau assembly works fine on a tiny corpus. I ran Queneau assembly ten times on ten samples from the spammy ebook titles, and here are the results:

  1. Beekeeping by Keep Grants
  2. Lose to Audience Business to to Your Backlink Physicists Environment
  3. HOT of Recruit Internet Because Financial the Memories
  4. Senior Guide Way! Business Way!
  5. Discover Can Power Successful Life How Steps
  6. Metal Lazy, Advice
  7. Insiders Came Warts Weapons Revealed
  8. 101 Secrets & THE Joint Health Than of Using Marketing! Using Using More Imagine
  9. Top **How Own 101**
  10. Multiple Spiritual Dynamite to Body - To Days

These are still really wacky, but they're better than when Queneau was choosing from 50,000 titles each time. For the @pony_strategies project, I still prefer the Markov chains.

Queneau wins when the outputs are short

Let's put spammy ebook titles to the side and move on to board game titles, a field where I think Queneau assembly is the clear winner. My corpus is here about 65,000 board game titles, gathered from BoardGameGeek. The key to what you're about to see is that the median length of a board game title is three words, versus nine words for a spammy ebook title.

Here are some of Markov's board game titles:

  1. Pointe Hoc
  2. Thieves the Pacific
  3. Illuminati Set 3
  4. Amazing Trivia Game
  5. Mini Game
  6. Meet Presidents
  7. Regatta: Game that the Government Played
  8. King the Rock
  9. Round 3-D Stand Up Game
  10. Cat Mice or Holes and Traps

A lot of these sound like real board games, but that's no longer a good thing. These are generic and boring. There are no surprises because the whole premise of Markov chains is that surprises are rare.

Here's Queneau:

  1. The Gravitas
  2. Risk: Tiles
  3. SESSION Pigs
  4. Yengo Edition Deadly Mat
  5. Ubongo: Fulda-Spiel
  6. Shantu Game Weltwunder Right
  7. Black Polsce Stars: Nostrum
  8. Peanut Basketball
  9. The Tactics: Reh
  10. Velvet Dos Centauri

Most of these are great! Board game names need to be catchy, so you want surprises. And short strings have highly ambiguous grammar anyway, so you don't get the "written by an alien" effect.

Conclusion

You know that I've been down on Markov chains for years, and you also know why: they rely on, and magnify, the predictability of their input. Markov chains turn creative prose into duckspeak. Whereas Queneau assembly simulates (or at least stimulates) creativity by manufacturing absurd juxtapositions.

The downside of Queneau is that if you can't model the underlying structure with code, the juxtapositions tend to be too absurd to use. And it's really difficult to model natural-language prose with code.

So here's my three-step meta-algorithm for deciding what to do with a corpus:

  1. If the items in your corpus follow a simple structure, code up that structure and go with Queneau.
  2. If the structure is too complex to be represented by a simple program (probably because it involves natural-language grammar), and you really need the output to be grammatical, go with Markov.
  3. Otherwise, write up a crude approximation of the complex structure, and go with Queueau.

[Comments] (2) Secrets of (peoples' responses to) @horse_ebooks—revealed!: As part of my @pony_strategies project (see previous post), I grabbed the 3200 most recent @horse_ebooks tweets via the Twitter API, and ran them through some simple analysis scripts to figure out how they were made and which linguistic features separated the popular ones from the unpopular.

This let me prove one of my hypotheses about the secret to _ebooks style comedy gold. I also disproved one of my hypotheses re: comedy gold, and came up with an improved hypotheses that works much better. Using these as heuristics I was able to make @pony_strategies come up with more of what humans consider the good stuff.

Timing

The timing of @horse_ebooks posts formed a normal distribution with mean of 3 hours and a standard deviation of 1 hour. Looking at ads alone, the situation was similar: a normal distribution with mean of 15 hours and standard deviation of 2 hours. This is pretty impressive consistency since Jacob Bakkila says he was posting @horse_ebooks tweets by hand. (No wonder he wanted to stop it!)

My setup is much different: I wrote a cheap scheduler that approximates a normal distribution and runs every fifteen minutes to see if it's time to post something.

Beyond this point, my analysis excludes the ads and focuses exclusively on the quotes. Nobody actually liked the ads.

Length

The median length of a @horse_ebooks quote is 50 characters. Quotes shorter than the median were significantly more popular, but very long quotes were also more popular than quotes in the middle of the distribution.

Capitalization

I think that title case quotes (e.g. "Demand Furniture") are funnier than others. Does the public agree? For each quote, I checked whether the last word of the quote was capitalized.

43% of @horse_ebooks quotes end with a capitalized word. The median number of retweets for those quotes was 310, versus 235 for quotes with an uncapitalized last word. The public agrees with me. Title-case tweets are a little less common, but significantly more popular.

The punchword

Since the last word of a joke is the most important, I decided to take a more detailed look each quote's last word. My favorite @horse_ebooks tweets are the ones that cut off in the middle of a sentence, so I anticipated that I would see a lot of quotes that ended with boring words like "the".

I applied part-of-speech tagging to the last word of each quote and grouped them together. Nouns were the most common by far, followed by verb of various kinds, determiners ("the", "this", "neither"), adjectives and adverbs.

I then sorted the list of parts of speech by the median number of retweets a @horse_ebooks quote got if it ended with that part of speech. Nouns and verbs were not only the most common, they were the most popular. (Median retweets for any kind of noun was over 300; verbs ranged from 191 retweets to 295, depending on the tense of the verb.) Adjectives underperformed relative to their frequency, except for comparative adjectives like "more", which overperformed.

I was right in thinking that quotes ending with a determiner or other boring word were very common, but they were also incredibly unpopular. The most popular among these were quotes that repeated gibberish over and over, e.g. "ORONGLY DGAGREE DISAGREE NO G G NO G G G G G G NO G G NEIEHER AGREE NOR DGAGREE O O O no O O no O O no O O no neither neither neither". A quote like "of events get you the" did very poorly. (By late-era @horse_ebooks standards, anyway.)

It's funny when you interrupt a noun

I pondered the mystery of the unpopular quotes and came up with a new hypothesis. People don't like interrupted sentences per se; they like interrupted noun phrases. Specifically, they like it when a noun phrase is truncated to a normal noun. Here are a few @horse_ebooks quotes that were extremely popular:

Clearly "computer", "science", "house", "and "meal" were originally modifying some other noun, but when the sentence was truncated they became standalone nouns. Therefore, humor.

How can I test my hypothesis without access to the original texts from which @horse_ebooks takes its quotes? I don't have any automatic way to distinguish a truncated noun phrase from an ordinary noun. But I can see how many of the @horse_ebooks quotes end with a complete noun phrase. Then I can compare how well a quote does if it ends with a noun phrase, versus a noun that's not part of a noun phrase.

About 4.5% of the total @horse_ebooks quotes end in complete noun phrases. This is comparable to what I saw in the data I generated for @pony_strategies. I compared the popularity of quotes that ended in complete noun phrases, versus quotes that ended in standalone nouns.

Quote ends in Median number of retweets
Standalone noun 330
Noun phrase 260
Other 216

So a standalone noun does better than a noun phrase, which does better than a non-noun. This confirms my hypothesis that truncating a noun phrase makes a quote funnier when the truncated phrase is also a noun. But a quote that ends in a complete noun phrase will still be more popular than one that ends with anything other than a noun.

Conclusion

At the time I did this research, I had about 2.5 million potential quotes taken from the Project Gutenberg DVD. I was looking for ways to rank these quotes and whittle them down to, say, the top ten percent. I used the techniques that I mentioned in my previous post for this, but I also used quote length, capitalization, and punchword part-of-speech to rank the quotes. I also looked for quotes that ended in complete noun phrases, and if truncating the noun phrase left me with a noun, most of the time I would go ahead and truncate the phrase. (For variety's sake, I didn't do this all the time.)

This stuff is currently not in olipy; I ran my filters and raters on the much smaller dataset I'd acquired from the DVD. There's no reason why these things couldn't go into olipy as part of the ebooks.py module, but it's going to be a while. I shouldn't be making bots at all; I have to finish Situation Normal.

@pony_strategies: My new bot, @pony_strategies, is the most sophisticated one I've ever created. It is the @horse_ebooks spambot from the Constellation Games universe.

Unlike @horse_ebooks, @pony_strategies will not abruptly stop publishing fun stuff, or turn out to be a cheesy tie-in trying to get you interested in some other project. It is a cheesy tie-in to some other project (Constellation Games), but you go into the relationship knowing this fact, and the connection is very subtle.

When explaining this project to people as I worked on it, I was astounded that many of them didn't know what @horse_ebooks was. But that just proves I inhabit a bubble in which fakey software has outsized significance. So a brief introduction:

@horse_ebooks was a spambot created by a Russian named Alexei Kouznetsov. It posted Twitter ads for crappy ebooks, some of which (but not all, or even most) were about horses. Its major innovative feature was its text generation algorithm for the things it would say between ads.

Are you ready? The amazing algorithm was this: @horse_ebooks ripped strings more or less randomly from the crappy ebooks it was selling and presented them with absolutely no context.

Trust me, this is groundbreaking. I'm sure this technique had been tried before, but @horse_ebooks was the first to make it popular. And it's great! Truncating a sentence in the right place generates some pretty funny stuff. Here are four consecutive @horse_ebooks tweets:

There was a tribute comic and everything.

I say @horse_ebooks "was" a spambot because in 2011 the Twitter account was acquired by two Americans, Jacob Bakkila and Thomas Bender, who took it over and started running it not to sell crappy ebooks, but to promote their Alternate Reality Game. This fact was revealed back in September 2013, and once the men behind the mask were revealed, @horse_ebooks stopped posting.

The whole conceit of @horse_ebooks was that there was no active creative process, just a dumb algorithm. But in reality Bakkila was "impersonating" the original algorithm—most likely curating its output so that you only saw the good stuff. No one likes to be played for a sucker, and when the true purpose of @horse_ebooks was revealed, folks felt betrayed.

As it happens, the question of whether it's artistically valid to curate the output of an algorithm is a major bone of contention in the ongoing Vorticism/Futurism-esque feud between Adam Parrish and myself. He is dead set against it; I think it makes sense if you are using an algorithm as the input into another creative process, or if your sole object is to entertain. We both agree that it's a little sketchy if you have 200,000 fans whose fandom is predicated on the belief that they're reading the raw output of an algorithm. On the other hand, if you follow an ebook spammer on Twitter, you get up with fleas. I think that's how the saying goes.

In any event, the fan comics ceased when @horse_ebooks did. There was a lot of chin-stroking and art-denial and in general the reaction was strongly negative. But that's not the end of the story.

You see, the death of @horse_ebooks led to an outpouring of imitation *_ebooks bots on various topics. (This had been happening before, actually.) As these bots were announced, I swore silent vengeance on each and every one of them. Why? Because those bots didn't use the awesome @horse_ebooks algorithm! Most of them used Markov chains, that most hated technique, to generate their text. It was as if the @horse_ebooks algorithm itself had been discredited by the revelation that two guys from New York were manually curating its output. (Confused reports that those guys had "written" the @horse_ebooks tweets didn't help matters--they implied that there was no algorithm at all and that the text was original.)

But there was hope. A single bot escaped my pronouncements of vengeance: Adam's excellent @zzt_ebooks. That is a great bot which you should follow, and it uses an approximation of the real @horse_ebooks algorithm:

  1. The corpus is word-wrapped at 35 characters per line.
  2. Pick a line to use as the first part of a tweet.
  3. If (random), append the next line onto the current line.
  4. Repeat until (random) is false or the line is as large as a tweet can get.

And here are four consecutive quotes from @zzt_ebooks:

Works great.

The ultimate genesis of @pony_strategies was this conversation I had with Adam about @zzt_ebooks. Recently my anger with *_ebooks bots reached the point where I decided to add a real *_ebooks algorithm to olipy to encourage people to use it. Of course I'd need a demo bot to show off the algorithm...

The @pony_strategies bot has sixty years worth of content loaded into it. I extracted the content from the same Project Gutenberg DVD I used to revive @everybrendan. There's a lot more where that came from--I ended up choosing about 0.0001% of the possibilities found in the DVD.

I have not manually curated the PG quotes and I have no idea what the bot is about to post. But the dataset is the result of a lot of algorithmic curation. I focused on technical books, science books and cookbooks--the closest PG equivalents to the crap that @horse_ebooks was selling. I applied a language filter to get rid of old-timey racial slurs. I privileged lines that were the beginnings of sentences over lines that were the middle of sentences. I eliminated lines that were boring (e.g. composed entirely of super-common English words).

I also did some research into what distinguished funny, popular @horse_ebooks tweets from tweets that were not funny and less popular. Instead of trying to precisely reverse-engineer an algorithm that had a human at one end, I tried to figure out which outputs of the process gave results people liked, and focused my algorithm on delivering more of those. I'll post my findings in a separate post because this is getting way too long. Suffice to say that I'll pit the output of my program against the curated @horse_ebooks feed any day. Such as today, and every day for the next sixty years.

Like its counterpart in our universe, @pony_strategies doesn't just post quotes: it also posts ads for ebooks. Some of these books are strategy guides for the "Pôneis Brilhantes" series described in Constellation Games, but the others have randomly generated titles. Funny story: they're generated using Markov chains! Yes, when you have a corpus of really generic-sounding stuff and you want to make fun of how generic it sounds by generating more generic-sounding stuff, Markov chains give the best result. But do you really want to have that on your resume, Markov chains? "Successfully posed as unimaginative writer." Way to go, man.

Anyway, @pony_strategies. It's funny quotes, it's fake ads, it's an algorithm you can use in your own projects. Use it!

November Film Roundup: What a month! Mainly due to a huge film festival, but I also got another chance to see my favorite film of all time on the big screen. What might that film be? Clearly you haven't been reading my weblog for the past fifteen years.

@everybrendan Season Two: Last year I wrote one of my first Twitter bots, @everybrendan. Inspired by Adam's infamous @everyword, it ran for two months, announcing possible display names for Brendan's Twitter account (background), taken from Project Gutenberg texts. Then I got tired of individually downloading, preparing, and scraping the texts, so I let it lapse a year ago today, with a call for requests for a "season two" that never materialized.

Well, season two is here, and it's a doozy. I've gone through Project Gutenberg's 2010 dual-layer DVD and found about 300,000 Brendan names in about 20,000 texts, enough to last @everybrendan until the year 2031. At that point I'll get whatever future-dump contains the previous twenty years of Project Gutenberg texts and do season three, which should keep us going until the Singularity. The season two bot announces each new text with a link, so it educates even as it infuriates.

I've been wanting to do this for a while, but it's a very tedious process to handle Project Gutenberg texts in bulk. Most texts are available in a wide variety of slightly different formats. The texts present their metadata in many different ways, especially when it comes to the dividing line between the text proper and the Project Gutenberg information. Some of the metadata is missing, some of it is wrong, and there's one Project Gutenberg book that doesn't seem to be in the database at all.

I started dealing with these problems for my NaNoGenMo project and realized that it wouldn't be difficult to get something working in time for the @everybrendan anniversary. I've put the underlying class in olipy: it's effectively a parser for Gutenberg texts, and a way to iterate over a CD or DVD image full of them. It can also act as a sort of lint for missing and incorrect metadata, although I imagine Project Gutenberg doesn't want to change the contents of files that have been on the net for fifteen years, even if some of the information is wrong.

The Gutenberg iterator still needs a lot of work. It's good enough for @everybrendan, but not for my other projects that will use Gutenberg data, so I'm still working on it. My goal is to cleanly iterate over the entire 2010 DVD without any problems or missing metadata. The problems are concentrated in the earlier texts, so if I can get the 2010 DVD to work it should work going forward.

[Comments] (3) Bots Should Punch Up: Over the weekend I went up to Boston for Darius Kazemi's "bot summit". You can see the four-hour video if you're inclined. I talked about @RealHumanPraise with Rob, and I also went on a long-winded rant that suggested a model of extreme bot self-reliance. If you take your bots seriously as works of art, you should be prepared to continue or at least preserve them once you're inevitably shut off from your data sources and your platform.

We spent a fair amount of time discussing the ethical issues surrounding bot construction, but there was quite a bit of conflation of what's "ethical" with what's allowed by the Twitter platform in particular, and website Terms of Service in general. I agree you shouldn't needlessly antagonize your data sources or your platform, but what's "ethical" and what's "allowed" can be very different things. However, I do have one big piece of ethical guidance that I had to learn gradually and through osmosis. Since bots are many hackers' first foray into the creative arts, it might help if I spell it out explicitly.

Here's an illustrative example, a tale of two bots. Bot #1 is @CancelThatCard. It finds people who have posted pictures of their credit or debit card to Twitter, and lets them know that they really ought to cancel the card and get a new one.

@CancelThatCard

Bot #2 is @NeedADebitCard. It finds the same tweets as @CancelThatCard, but it retweets the pictures, collecting them in one place for all to see.

@NeedADebitCard

Now, technically speaking, @CancelThatCard is a spammer. It does nothing but find people who mentioned a certain phrase on Twitter and sends them a boilerplate message saying "Hey, look at my website!" For this reason, @CancelThatCard is constantly getting in trouble with Twitter.

As far as the Twitter TOS are concerned, @NeedADebitCard is the Gallant to @CancelThatCard's Goofus. It's retweeting things! Spreading the love! Extending the reach of your personal brand! But in real life, @CancelThatCard is providing a public service, and @NeedADebitCard is inviting you to steal money from teenagers. (Or, if you believe its bio instead of its name, @NeedADebitCard is a pathetic attempt to approximate what @CancelThatCard does without violating the Twitter TOS.)

At the bot summit I compared the author of a bot to a ventriloquist. Society allows a ventriloquist a certain amount of license to say things via the dummy that they wouldn't say as themselves. I know ventriloquism isn't exactly a thriving art, but the same goes for puppets, which are a little more popular. If you're an MST3K fan, imagine Kevin Murphy saying Tom Servo's lines without Tom Servo. It's pretty creepy.

We give a similar license to comedians and artists. Comedians insult audience members, and we laugh. Artists do strange things like exhibit a urinal as sculpture, and we at least try to take them seriously and figure out what they're saying.

But you can't say absolutely anything and expect "That wasn't me, it was the dummy!" to get you out of trouble. There is a general rule for comedy and art: always punch up, never punch down. We let comedians and artists and miscellaneous jesters do outrageous things as long as they obey this rule. You can poke fun at yourself (Stephen Colbert famously said "There's no status I would not surrender for a joke"), you can make a joke at the expense of someone with higher social status than you, but if you mock someone with lower status, it's not cool.

If you make a joke, and people get really offended, it's almost certainly because you violated this rule. People don't get offended randomly. Explaining that "it was just a joke" doesn't help; everyone knows what a joke is. The problem is that you used a joke as a means of being an asshole. Hiding behind a dummy or a stage persona or a bot won't help you.

@NeedADebitCard feels icky because it's punching down. It's saying "hey, these idiots posted pictures of their debit cards, go take advantage of them." Is there a joke there? Sure. Is it ethical to tell that joke? Not when you can make exactly the same point without punching down, as @CancelThatCard does.

The rules are looser when you're in the company of other craftspeople. If you know about the "Aristocrats" joke, you'll know that comedians tell each other jokes they'd never tell on the stage. All the rules go out the window and the only thing that matters is triggering the primal laughter response. But also note that the must-have guaranteed punchline of the "Aristocrats" joke ensures that it always ends by punching upwards.

You're already looking for loopholes in this rule. That's okay. Hackers and comedians and artists are always attracted to the grey areas. But your bot is an extension of your will, and if you're a white guy like me, most of the grey areas are not grey in your favor.

This is why I went through thousands of movie review blurbs for @RealHumanPraise in an attempt to get rid of the really sexist ones. It's an unfortunate fact that Michelle Malkin has more influence over world affairs than I will ever have. So I have no problem mocking her via bot. But it's really easy to make an incredibly sexist joke about Michelle Malkin as a way of trying to put her below me, and that breaks the rule.

There was a lot of talk at the bot summit about what we can do to avoid accidentally offending people, and I think the key word is 'accidentally.' The bots we've created so far aren't terribly political. Hell, Ed Henry, chief White House correspondent for FOX News, follows @RealHumanPraise on Twitter. If he enjoys it, it's not the most savage indictment.

In comedy terms, we botmakers are on the nightclub stage in the 1950s. We're creating a lot of safe nerdy Steve Allen comedy and we're terrified that our bot is going to accidentally go off and become Andrew Dice Clay for a second. There's nothing wrong with Steve Allen comedy, but I'd also like to see some George Carlin type bots; bots that will, by design, offend some people. (Darius's @AmIRiteBot is the only example I know of.)

Artists are, socially if not legally, given a certain amount of license to do things like infringe on copyright and violate Terms of Service agreements. If you get in trouble, the public will be on your side, unless you betrayed their trust by breaking the fundamental ethical rule of comedy. So do it right. Design bots that punch up.

In Dialogue: I wanted to participate in Darius Kazemi's NaNoGenMo project but I already have a novel I have to write, so I didn't want to spend too much time on it. And I did spend a little more time on this than I wanted, but I'm really happy with the result.

"In Dialogue" can take all the dialogue out of a Project Gutenberg book and replace it with dialogue from a different book. My NaNoGenMo entry is in two parts: "Alice's Adventures in the Whale" and "Through the Prejudice Glass".

You can run the script yourself to generate your own mashups, but since there are people who read this blog who don't have the skill to run the script, I present a SPECIAL MASHUP OFFER. Send me email or leave a comment telling me which book you want to use as the template and which book you want the dialogue to come from. I'll run the script for you and send you a custom book.

Restrictions: the book has to be on Project Gutenberg and it has to use single or double quotes to denote dialogue. No continental chevrons or fancy James Joyce em-dashes. And the dialogue book has to be longer than the template book, or at least have more dialogue.

: Last week I had a little multiplayer chat with Joe Hills, the Minecraft mischief-maker. The result is a two-part video on Joe's YouTube channel: part 1, part 2. Our main topic of conversation was the antisocial, self-destructive things creative people do, and how much of that is actually tied to their creativity.

I should have posted this earlier so I could have said "I dreamed I saw Joe Hills last night," but that's life.


This document (source) is part of Crummy, the webspace of Leonard Richardson (contact information). It was last modified on Monday, September 09 2013, 18:05:52 Nowhere Standard Time and last built on Tuesday, July 22 2014, 21:20:03 Nowhere Standard Time.

Crummy is © 1996-2014 Leonard Richardson. Unless otherwise noted, all text licensed under a Creative Commons License.

Document tree:

http://www.crummy.com/
Site Search: