blags by matthttp://blagomatic.com/b/matt/blagsblagomatic - genetic gestureslearn. do. fun.http://matt.io/technobabble/learn.__do.__fun./uvhttp://learndofun.com/ -- what did you do?

Here's an edited-out excerpt from learndofun prelaunch ranting about existing online educational garbage:

They've taken the format of university lectures, removed the whiteboard, scaled down a writing surface to the size of two hands, and they make you watch as they write out everything. If you try to watch these videos on a large screen, the hand is like some freaky spider of knowledge spinning out ink in droning patterns. There's a voice to the video, but no human interaction. No face. No human level contact. You are basically watching them write a textbook. A very poorly formatted and untypeset textbook.


We can do better. We will do better.]]>
Tue, 08 Jan 2013 12:42:47 GMThttp://matt.io/technobabble/learn.__do.__fun./uv
sleepwalking into nightmareshttp://matt.io/found_things/sleepwalking_into_nightmares/ut
"The way in which people will give away their personal details in much the way as in the child history books about colonialism. You had native chieftains who handed over mineral rights in return from some baubles from some canny imperialist. All that stuff is happening but it's very hard to explain to anybody."

(At the 36 minute mark = http://www.youtube.com/watch?v=yUXh-GPa5dI&t=36m00 )



Thanks to Stephen for pointing me towards the video.]]>
Mon, 21 Nov 2011 12:15:06 GMThttp://matt.io/found_things/sleepwalking_into_nightmares/ut
I don't think I'm going to go to LA anymorehttp://matt.io/found_things/I_don't_think_I'm_going_to_go_to_LA_anymore/us
Original Mass Performance. Where The Light Is: 2007.


More Recent Mass Performance. BJCC: 2010.


Unbelievably good cover


Covered by an adorable guy

]]>
Wed, 09 Nov 2011 11:59:29 GMThttp://matt.io/found_things/I_don't_think_I'm_going_to_go_to_LA_anymore/us
hivemind devops alert: nginx does not suck at sslhttp://matt.io/technobabble/hivemind_devops_alert:_nginx_does_not_suck_at_ssl/urupdate
Hot on the heels of hivemind devops alert: nginx sucks at ssl, I issue another hivemind devops alert: nginx does not suck at ssl!

aftermath
After circulating my previous alert through the 'tubes, it became abundantly clear: nobody has any idea how SSL performance works. A few people suggested I was running out of entropy (nice guess, but wrong), many people mentioned ssl session caching (nice try, but not relevant when testing all new connections), and a few people chimed in about keepalive (nice try, but then results get skewed depending on how many assets each client requests). Everybody seemed to care about the absolute numbers and not the relative performance differences.

A few people went with a very tactful "dude, that's just wrong. I know it works" response which is perfectly valid and appreciated. I knew something was wrong, but couldn't put my finger on it.

Then David chimed in, and it clicked. I wasn't verifying equal cipher negotiation against all servers and the benchmark utility. Even so, why would nginx pick a more computationally intensive cipher than stunnel or stud? Let's find out.

the what
The problem is that annoying line in your nginx config you copied from somewhere else and you're not entirely sure what it does. It's a lengthy security incantation. Maybe you picked a PCI compliant list? Maybe you filtered it yourself? It looks something like [a]:

ssl_ciphers ALL:!aNULL:!ADH:!eNULL:!MEDIUM:!LOW:!EXP:RC4 RSA: HIGH;


So what does it mean? The line above gives you a nice and secure encryption setup for nginx. Unfortunately, it also includes a very computationally intensive cipher using an ephemeral Diffie-Hellman exchange for PFS. Sounds scary already, doesn't it?

the how
The problem cipher is DHE-RSA-AES256-SHA [b]. It uses DH on each new connection to ensure PFS (DHE = "Diffie-Hellman Ephemeral"). The DH portion requires extra code in software using OpenSSL to enable DHE negotiation. nginx has the extra code built in. stud doesn't have at all. stunnel has it as a compile time/certificate configurable option.

Ding!

nginx configures OpenSSL so completely, it enables the very-secure-yet-very-slow cipher by default! stunnel does not enable that cipher by default because it doesn't configure DH itself (you can configure it by hand though). stud does not enable DH at all.

The next-most-secure cipher is AES256-SHA, which is what stunnel and stud were using to out-perform nginx on my thundering herd connection tests.

the fix
You can force nginx to not enable the expensive cipher by excluding all DHE ciphers. Add "!kEDH" to your cipher list. It disables (the ! disables) any cipher using Ephemeral Diffie-Hellman.

the new nginx benchmarks
Now that we found the problem, let's look at nginx SSL using a sane performance cipher.

nginx (AES256-SHA) -> haproxy: 1300 requests per second
nginx (AES256-SHA with keepalive 5 5;) -> haproxy: 4300 requests per second


There is a slight speed boost from also disabling iptables. As always, do not trust these numbers. Performance depends on: your firewall config, your ciphers, your backend latency, keepalive, session caching, and how many faeries currently live in your system fans [c].

final answer
To get more performance out of nginx SSL, remove its ability to negotiate slow encryption ciphers. Add "!kEDH" to your list of allowed ciphers (unless you are passing around government secrets about aliens or are an internationally wanted arms dealer). Do it now.

Curious about what cipher your install is negotiating? Test it with a quick:

openssl s_client -host HOSTNAME -port 443

Look at the Cipher: line. If it says DHE-RSA-AES256-SHA, your site could be going much faster over SSL by disabling DHE.


-Matt

[a]: If you want to understand the format (! versus versus -), read http://www.openssl.org/docs/apps/ciphers.html.

[b]: "problem" from a website speed point of view.

[c]: Disable iptables on all web-facing boxes if you want to maximize performance. Use SSL session caching and (sane) keepalive values, but those settings aren't panaceas. Everything depends on your page composition, user habits, and individual system issues.


bonus insight about SSL benchmarks
Don't trust anybody's SSL benchmarks unless they include at a minimum details about: Is the OS firewall enabled? Is the benchmark being run over localhost on the same machine? Which cipher is being negotiated between the benchmark tool and the SSL server? Is keep alive on? Is the benchmark tool using keep alive? Is session resumption on? Is the benchmark tool using session resumption? Which benchmark program is being used (they all have different inherent performance problems)?

(note: I didn't mention any of those things. Do not trust my numbers. Benchmark your own systems.)

Over-reliance on becnhmarking keepalive and session resumption can yield false results unless you ever only have one client to your website and they use one keepalive connection and one SSL session constantly.

If you care about absolute numbers, require details about: How many cores? How fast? Do you have an OpenSSL-recognized hardware accelerator engine being used? What else is running on the box? What's the latency among all components?

bonus insight about social diarrhea
I launched my original post over the HN fence and to my twitter account at the same time. It quickly fell off the new page of HN. It immediately started getting re-twatted on the twitters.

Every @reply I received from twitter was supportive, helpful, understanding, or very politely confused/questioning.

Later in the day, to stop me from whining, a friend re-submitted my post to HN. This time the article shot to the #1 spot. Uh oh. If I hadn't developed such think Internet Defense skin over the years, I would have been terribly offended by half of the HN comments.

Remember: The Internet is a big place. If you get upset when somebody isn't as perfect as you are, you'll spend your life being miserable. Be nice. Be understanding.

Final feeling: Twitter is better than HN in all social dimensions of engagement, kindness, and authenticity.

fin.]]>
Wed, 13 Jul 2011 23:10:44 GMThttp://matt.io/technobabble/hivemind_devops_alert:_nginx_does_not_suck_at_ssl/ur
hivemind devops alert: nginx sucks at sslhttp://matt.io/technobabble/hivemind_devops_alert:_nginx_sucks_at_ssl/uqUPDATE
The problem with nginx is resolved in nginx does not suck at ssl!

background
What do you use to serve content over SSL? mod_ssl? nginx compiled with ssl support? stunnel? A hardware accelerator-jigger?

I benchmarked a few SSL terminators in front of haproxy last week. The results may (or may not) surprise you.

initial benchmark results

(on an 8 core server...)
haproxy direct: 6,000 requests per second
stunnel -> haproxy: 430 requests per second
nginx (ssl) -> haproxy: 90 requests per second


initial benchmark results reaction
what. the. fuck.

<rhetorical>
Why is nginx almost 5 times slower than stunnel? It can't all be nginx's http processing, can it? What is crappifying nginx's SSL performance? [a]
</rhetorical>

After recovering from the shock of nginx's crap ssl performance and cleaning up spewed hot chocolate off my monitor, stud strutted up to me and begged to be benchmarked too (he hates being left out). stud looks perfect -- a simple, uncrapified TLS/SSL terminator created because stunnel is old and bloated.

stud's one glaring fault is a lack of HTTP header injection support for adding X-Forward-For headers. [1]

Woe is me. How do we get around not having X-Forward-For headers? Do we sit around and complain online? Do we pay someone else to add it? Do we stay with nginx because it's "what we know?" Heck no. Write it yourself.

Now we have a stud with X-Forward-For support. [2]

More benchmarks against plain stud (factory default) and stud with http header injection added:

more benchy markys

(on the same 8-core server...)
first, results from before:
haproxy direct: 6,000 requests per second
stunnel -> haproxy: 430 requests per second
nginx (ssl) -> haproxy: 90 requests per second

now, enter stud (the -n number is how many cores are used):
stud -n 8 -> haproxy: 467 requests per second
stud-jem -n 8 -> haproxy: 475 requests per second

stud-http-jem -n 1 -> haproxy: 440 requests per second
stud-http-jem -n 7 -> haproxy: 471 requests per second
stud-http-jem -n 8 -> haproxy: 471 requests per second


We have a winner! (special note: according to my tests, running stud with jemalloc speeds it up in all cases.)

The added work of parsing, extracting bad headers, and injecting proper ones shows no practical performance impact versus factory default stud.

okay, so what did you do?
I've modified the crap out of stud and its Makefile. All changes are sitting in my add-HTTP-x-forward-for branch on my le github.

Modifications to stud so far:
Dependencies (libev, http-parser, jemalloc) automatically download during the build process. Nothing needs to be installed system-wide.

I cleaned up the build process for stud so you can configure it in a dozen different ways without rewriting the entire Makefile.

By default, everything is statically linked. You can move your one stud binary to another server without installing libev or jemalloc. [3]

The stud Makefile now builds four binaries (if you "make all"): stud, stud-jem, stud-http, and stud-http-jem. jem means "with jemalloc" and http means "automatically injects X-Forward-For and X-Forward-Proto headers." [4]

All http support is isolated in ifdef blocks. Running a non-http stud is exactly the same as stud from bumptech/stud.

in short
<LIES>
Use stud. Don't use stunnel. Never let nginx listen for SSL connections itself.
</LIES>

<TRUTH>
Keep using nginx for SSL termination. Just make sure your ciphers are set correctly. See nginx does not suck at ssl for an overview of how to fix your nginx config and why this post is wrong.
</TRUTH>


-Matt, your friendly bay area neighborhood web performance junkie.

[a]: I tested nginx as a proxy, serving static files, and serving nginx-generated redirects. I tried changing all the relevant ssl parameters I could find. All setups resulted in the same SSL performance from nginx. I even tried the setup on more than one server (the other server was quad-core nginx got up to 75 requests per second).

[1]: Yes, stud supports writing source IP octets before a connection and now even writing haproxy's own source format before a connection, but I like routing everything back through nginx.

[2]: note: still in-progress. It works, but you can probably craft bloated headers to drop your connection (stud won't crash or segfault -- the errors just break the connection).

[3]: At the top of the Makefile you can easily twiddle static linking on and off per library (for libev and/or jemalloc).

[4]: X-Forward-For header injection is done properly. Any X-Forward-For or X-Forward-Proto headers originating from the client are removed and then replaced by stud-injected headers. We can't allow our clients to inject X-Forward headers our applications expect to be truthful.]]>
Mon, 11 Jul 2011 12:02:13 GMThttp://matt.io/technobabble/hivemind_devops_alert:_nginx_sucks_at_ssl/uq
It's now my 25th hour with scalpel in handhttp://matt.io/found_things/It's_now_my_25th_hour_with_scalpel_in_hand/up
Radio


Letter From My Lonelier Self


Everything's Fine


Okay, New York (Gonna Make it Home)


Promises


Stray Italian Greyhound



Blue Caravan



It's Not Even Christmas, It's Just Thursday Night (Christmas Song 2010)


]]>
Mon, 27 Dec 2010 06:16:48 GMThttp://matt.io/found_things/It's_now_my_25th_hour_with_scalpel_in_hand/up
It's Open Source, Bitchhttp://matt.io/technobabble/It's_Open_Source,_Bitch/uo
You don't matter. Your holy of holy open source principles don't matter. The users matter.

I "upgraded" from Fedora 13 to 14 recently. Well, I tried to upgrade. The upgrade failed using their special upgrade installer. I resorted to a clean install instead.

End result: less functionality.

Flash stopped working. Drag and drop stopped working. My (very common) video driver didn't work out of the box and required waiting for an update from Fedora weeks after the GM.

Even with the fixes so far, video is more choppy than before the upgrade. Drag and drop still doesn't work. Audio seems to have a mind of its own, but for now I'll blame audio problems on Chrome vs. nspluginwrapper vs. Flash vs. pulseaudio.

What's going on with Fedora?

Their bug tracker sheds light on how screwed up the Fedora maintainers can be: https://bugzilla.redhat.com/show_bug.cgi?id=638477

The short version: someone notices Flash isn't working properly. They track it down to a glibc problem. glibc closes it on their side as NOTABUG because technically, Flash is using memcpy incorrectly. A recent update to glibc made an optimization breaking overlapping memcpy calls. Even knowing 64 bit Flash isn't working, Fedora goes ahead with the release. As of a few weeks after the release, 64 bit Flash still doesn't work without a manual compile-your-own-shared-driver hack.

Why should I continue to use Fedora as a desktop OS if it refuses to test functionality of the most common desktop software program in the wild?

Personalities in the bug thread fall in to four categories: problem reporter, maintainer/admin, helpful guru, and peanut gallery (ideologues). It seems the ideologues and admins are the same group though.

The Fedora maintainer running the thread chimes in, "The only stupidity is crap software violating well known rules that have existed forever."

The official position of Fedora is to ship a disto with broken functionality because it's the "crap software violating well known rules." Screw the users, their software vendor is shipping buggy software. But, the bug never showed up until our most recent OS release.

Linus calmly chimes in when faced with the rude outburst, saying he understands the software is broken, but asks "What's the upside of breaking things? That's all I'm asking for."

Linus declares, "Are you seriously going to do a Fedora-14 release with a known non-working flash player?"

Yes, Fedora shipped a release with broken 64 bit Flash player functionality. They don't seem to care.

Dan reiterates what Linus is trying to say: "For a CLOSED NOTABUG bug report, seems to be a lot of traffic on it. Is ANYONE actually fixing this problem either in glibc or in Flash? This is ridiculuous."

Michael agrees: "If it works on Win 7 and doesn't work on Fedora, it is Fedora and Linux that take the crap. Period."

Linus continues to make points the Fedora maintainers ignore and refuse to comment about:
"Rather than make it look bad in the eyes of users who really don't care _why_ flash doesn't work, they just see it not working right.

There is no advantage to being just difficult and saying 'that app does something that it shouldn't do, so who cares?'. That's not going to help the _user_, is it?

And what was the point of making a distro again? Was it to teach everybody a lesson, or was it to give the user a nice experience?"

The same autistic-absolute-right-ideology versus sanity is repeated in a different thread at fedorahosted.


I haven't even touched on my other two issues. Drag and drop is broken. A feature from 1980 doesn't work in 2010. My video card, using the most used video driver on desktop Linux, didn't work in the release without a manual Fedora software update. Video card reason? It's using a proprietary driver so they don't test it.

What is Fedora doing?
]]>
Fri, 26 Nov 2010 16:07:57 GMThttp://matt.io/technobabble/It's_Open_Source,_Bitch/uo
The Key-Value Wars of the Early 21st Centuryhttp://matt.io/technobabble/The_Key-Value_Wars_of_the_Early_21st_Century/ui
"Give us replication or... give us an acceptable alternative!"

"If I have to write another schema migration, I swear to Monty, I'm going to become a hardware store day labourer."

"Serialized JSON is the answer to everything!"
"But what if I want to search by a property of the JSON object and not just the id?"
"You just write a map-reduce function and re-reduce until you get your answer."
"Did you just tell me to go computationally fuck myself?"
"I believe I did, Bob. I believe I did." [q]

The core of their debate came down to representation versus distribution. The squealers claimed to have perfect representation, but hacked together distribution methods. The kvalers claimed to have perfect distribution, but limited representation models.

Trees sitting next to tables
What was the problem with representation? Why is perfect representation diametrically opposed to perfect distribution? Perfect representation came down to one issue: asking for ranges of things. Who is between 18 and 29? Which orders completed between last Thursday and today?

The squealers worshiped at the altar of their B+ tree, offering up mathematically perfect schemas in return for logarithmic access times to their data. Squealers ignored the one downside of their Lord of All Data Structures: B+ trees were manipulated in-place on disk, requiring huge amounts of logically contiguous disk space for large datasets. Distributing and replicating a B+ tree across multiple machines was impossible. Squealers took to chopping up their data into smaller and smaller collections until entire copies of B+ trees could be kept on multiple machines. Squealers claimed they never needed needed cross-table joins in the first place and they actually enjoyed the additional administration of dealing with statement based replication [a].

The kvalers worshiped at the altar of their hash table, offering up non-colliding keys in return for separation of data enabling perfect distribution and replication. The kvalers ignored the one downside of their Lord of All Data Structures: hash tables provide no data locality. Kvalers happily maintained hand-crafted indexes if range queries or search-within-the-dataset queries were required. Kvalers pretended to never have heard of ad-hoc queries or business intelligence requirements.

Who put my tree on the table?
Trees are made of nodes. Nodes must have names to be found. Traditionally, B+ trees were stored in files with nodes being referenced by positions within the B+ tree file. Arbitrarily chopping a B+ tree in half and distributing it to another node was not practical. Even with a B+ tree designed to reference outside its own file [1], the entire collection of mini-B+ trees needed to be present for operations to complete, resulting in no advantage in distribution or replication.

The B+ tree scalability problem came down to one issue: how can you name a node and later reference it independent of its storage location?

One mild summer's day in Goolgonia, historically California, someone stumbled upon the answer: a new way of naming nodes [2]. What if, instead of offsets within a file, a Central Naming Service could provide a globally unique node identifier serving as a recall key for a node's location? Instead of storing nodes within a sequentially allocated data file, B+ tree nodes could be stored on anything accepting the name of the node and storing its contents.

The kvalers squealed.

The squealers begged the kvalers, "Please, may we have access to your infinite-storage-capacity, automatically-redundant-with-self-managing-failover, and incorruptible storage system?" The response was delivered with a wry smile: "We've been waiting for you."

In a sigh of ecstasy, the squealers and kvalers realized what was born. The squealer's toes curled at the thought of being free from routine sharding, free from identifying hot spots and spinning off copies, free from arbitrarily chopping tables in half, and free from single points of failure. The kvaler's eyes rolled upwards while synthesizing thoughts of sequential data access, retrieving records by ranges, and iterating over database snapshots just by holding on to a root node.

After an exhaustive journey, the kvalers and squealers drifted off to a sound sleep that night. The story of the squealers versus kvalers came to a close, but the journey of the sqvalers had just begun.

wtfbbq? (aka Back to Life, Back to Reality)
Over the past few years I've been looking for a stable, low maintenance, zero up front decision making, scalable sorted data platform. I looked high and low to no avail. I even went as far as writing a single-purpose BigTable clone called TinyTable for my own use, but it still had annoying edge cases when I overflowed a single disk or RAID array.

A potential solution dawned on me after looking at an R-tree implementation using CouchDB's append-only storage format. To store a node, you pass in the data and get back its position. To retrieve a node, you pass in a position and get back the data. It's that simple.

All file writing and reading is completely encapsulated within the couch_file module. The two storage calls are straight forward. No seeking, block management, or any other file-level details are leaked through the interface.

{ok, Position} = couch_file:append_term(Fd, Node).
{ok, Node} = couch_file:pread_term(Fd, Position).


Important point: Position is an opaque type. The code doesn't care if Position is an integer or a tuple or a binary. Position is passed through to the storage layer on lookups and returned to the caller on appends.

Let's do this
To make CouchDB store documents remotely, we only have to replace the implementation of the two functions listed above. For our remote storage let's use Riak as our Key-Value store (because it's awesome). CouchDB persists Erlang terms to disk and Riak persists Erlang terms to disk. We get to remove redundant code from CouchDB since Riak is converting terms for us. Riak also automatically replicates everything we store, easily handles adding more machines to increase capacity, and deals with failures transparently.

append_term's implementation becomes riak_write(Bucket, Key, Value).
pread_term's implementation becomes riak_read(Bucket, Key).
We replaced Fd with a Bucket name here. Bucket is the database filename as if it were stored locally (e.g. <<"_users.couch">>).

We also have to generate unique keys for our nodes. The CouchDB file format is append-only, so we never have to worry about updating a value once it is stored [e]. Where does Key come from? Key is equivalent to Position in the original couch_file calls. Key is now simply {node(), now()}. That's the Erlang way of making a globally unique value within your cluster. node() is the name of your VM (original in a cluster) and now() returns the current time (guaranteed to be monotonically increasing) [k].

We turn our nice {node(), now()} into a binary and use it as a key: term_to_binary({node(), now()}), et voila we have a globally unique key to store values in Riak [m].

All the tricky couch_file page alignment code is now gone. The only files written to the local file system are couch.uri and couch.log.

We did this
Does it work? Sure, it works. Check out my CouchDB Riak Integration for yourself [p]. Modify your bin/couchdb script to include Riak client libraries (riak_kv) and connect to a local Riak cluster:
-pa ../riak/deps/riak_kv/ebin -pa ../riak/deps/riak_core/ebin -setcookie riak -name 'couchdb@127.0.0.1'
Insert those after $ERL_START_OPTIONS and before -env ERL_LIBS

What did it take to convert CouchDB to use completely remote storage for documents? It took: 7 files changed, 87 insertions(+), 502 deletions(-)

See the commit log for quirks, caveats, and how to use the admin interface properly in my proof-of-concept implementation.

Patches welcome? Maybe? To make this mergeable upstream, we need a way to have both local-disk and Riak storage. The original couch_file needs to be restored with flags about when to use remote storage versus local storage per-DB. A dozen other quirks and deficiencies would have to be fixed and accounted for as well. It makes more sense to use the CouchDB B+ tree code (and/or the R-tree code) with a modified couch_file to create an independent data platform [L].


-Matt
@mattsta

You shouldn't follow @mattsta on twitter.
He only writes about annoying problems, trivialities of life, The Bay Area, and programming.
Pretty boring stuff.


Footnotes:

[q]: With apologies.

[a]: Yes, this is the shitty MySQL way of doing things. Let's not tell them about PostgreSQL's log shipping replication.

[1]: e.g. Replacing your node reference from {Offset} to {Filename, Offset} still means all referenced files must be replicated.

[2]: ANWONN: The most advanced, world-changing, node naming scheme to ever be conceived by an almost god-like being. Do you question it? Here, read a 600 page autobiography about how amazing, unique, and intelligent I am.

[3]: This provides more location independence than the the Sequential Append-Only Write-Once-Read-Many Just-Try-To-Corrupt-Me-I-Dare-You B+ tree file format. [[Yes, this footnote isn't referenced in the article above. I'm not sure when the source to the footnote got cut, but I like the footnote anyway.]]

[k]: Yeah, I just re-wrote Twitter's Snowflake in one line of Erlang. And we can deal with unsigned longs too because we're Not Java. Let's ignore the fact that {node(), now()} is about 300 bits instead of 64 bits.

[m]: Yes, you can (and should) make it much more efficient storage-wise. I'm going for concise and readable here.

[p]: The CouchDB code tree is very messy. It really needs to be cleaned up. I wish they would discover Rebar already. I don't give a shit about the bulk of it, still, I keep it professional.

[L]: I'm not a fan of storing documents as JSON.

[e]: Except for the header storing root node information.
]]>
Mon, 09 Aug 2010 22:12:37 GMThttp://matt.io/technobabble/The_Key-Value_Wars_of_the_Early_21st_Century/ui
nodejs -> twitter -> nodejs -> couchdbhttp://matt.io/technobabble/nodejs_->_twitter_->_nodejs_->_couchdb/uhnode client to import the public twitter timeline into couchdb running at http://127.0.0.1:5984/ (create the database at /twitter/ first).

Everybody can run 150 public API requests per hour which nets you around 3000 tweets per hour. The client polls the public API every 0.5 seconds.

Runs against 2010.07.16 node-v0.1.101.


#!/usr/bin/env node

var http = require('http');
var couch = http.createClient(5984, '127.0.0.1');
var twitter = http.createClient(80, 'api.twitter.com');

function post_all_json(twitter_json_text) {
var creq = couch.request('POST', '/twitter/_bulk_docs',
{'content-type': 'application/json'});
creq.write('{"docs":' + twitter_json_text + '}');
creq.end();

}

function runTwitterRequest() {
var treq = twitter.request('GET', '/1/statuses/public_timeline.json',
{'host': 'api.twitter.com'});
treq.end();
treq.on('response', function(response) {
var twitter_response = "";
response.setEncoding('utf8');
response.on('data', function(chunk) {
twitter_response += chunk;
});
response.on('end', function() {
var tjson = JSON.parse(twitter_response);
console.log("Posting retrieved tweets which number " + tjson.length);
post_all_json(twitter_response);
});
});
}

setInterval(runTwitterRequest, 500);

]]>
Sun, 25 Jul 2010 03:48:57 GMThttp://matt.io/technobabble/nodejs_->_twitter_->_nodejs_->_couchdb/uh
redis meetup noteshttp://matt.io/technobabble/redis_meetup_notes/uf
libcluster
Redis is getting native clustering one way or another. What if we make a generic libcluster to handle the hash ring, membership, networking, failover and other clustery things? It could be helpful in other projects if libcluster is as well thought out and as well written as Redis.

zeromq
How about adding zeromq as a connection option in addition to TCP and (soon) unix domain sockets? Zeromq can give us sane udp out of the box (maybe) and possibly a more efficient tcp layer since zeromq coalesces as many outgoing messages together as possible (nb: would break a central NIH rule of antirez).

command response tags
Someone brought up the issue of multiple threads using one redis connection. What if each redis command could accept an optional tag that got echoed back in the response to the user? Client libraries can make a tag (something unique within a short time period of the app), send commands to redis as needed, then use the tags in responses to match up redis responds with who should receive them.

Sounds good to me.

-Matt ]]>
Thu, 24 Jun 2010 15:07:56 GMThttp://matt.io/technobabble/redis_meetup_notes/uf
Work at a Startup Reviewhttp://matt.io/startup/Work_at_a_Startup_Review/ud
One after another in sternly administered choreography, the startups gave brief pitches to the attendees. Pitches consisted of projected slides, one or more of the startup founders presenting an employee-focused view of their company, and endless fidgeting with a shared lavalier. Every few pitches, a person comfortable with public speaking became slightly more engaging by including a developer-friendly joke.

Notable presentations
Drew Houston of Dropbox received a vocal "Woo!" when he introduced himself. Everybody loves The DropBox.

The pas de deux between a seemingly nervous Sam Altman talking quietly to his slides and Paul Graham in the back holding up his trademark "LOUDER!" sign entertained the crowd for a full 60 seconds. It turns out you can't see Paul Graham holding up a sign, no matter how bright his shirt, if you are looking at a wall. (Don't worry Sam, I still love you.)

The presenter for airbnb ("The Abees") was unquestionably the most direct, forceful, and well rehearsed speaker of the event. He knows what airbnb wants, how they are going to get there, how awesome they are, how much the world suffers, and why they are worth more than you can possibly imagine.

Paul S joined ex-Twitter employee Chad to talk about notifo (pronounced note-ee-foe, but which reminds one of Jeff's notify.io). Paul, being with notifo for about two days, talks as if he worked on the project for months. You can't out plan, out hipster, or out self-promote Paul S.

Reception
After the second "this is the last presentation" presentation, the attendees were invited out to the parking-lot-turned-pizza-buffet for a few hours of mingling, scheming, catching up, and getting to know one another. Startup founders darted through the crowd as little entrepreneurial sharks swimming through the chum of potential employees. Pairs of platonic speed dates broke off. Some pairs chatted for half an hour while others lasted barely thirty seconds. Some people greeted with a firm handshake while others only yielded a sideways glance of yes-i-see-you-but-don't-talk-to-me.

More interesting than talking with the intended audience of startup representatives was talking with fellow attendees. You can see in their eyes when they are good at what they do. You can see in their eyes when they are uncertain what tomorrow brings. You can see the crazy ones, the dreamers, and the failures. At least one attendee did not know what Y Combinator actually did. Other attendees ranged from Lockheed Martin employees, to in-between-semester undergrads, to i'm-taking-this-loop-off people taking a break from being employees, to people just scouting for a slightly more interesting life.

Notable reception happenings
One person was carrying a copy of On Lisp. I assume he obtained a useful signature.

The venerable Sam Altman, the chosen one, the golden child of Paul Graham, said he likes my HN comments. He then proceeded to tell me the story of how he almost killed someone then escaped death at the hands of mother nature. I've stories for you too, Sam. Give me a call.

[[This brings me to another point: name badges are for names. We all have multiple names. I was the only one I saw with my Human, HN and Twitter names on my badge. Come on people, use a blank canvas more creatively. We want to know you.]]

I thought Trevor said the telepresence bots would be active, but they never emerged. A row of bots lined up obediently at the Anybots entrance waiting for their chance to be of service. Their chance never came.

Paul Graham attracted his large conversational following as usual. I didn't notice Trevor emerge outside. Kate gave me a nod but didn't want to talk. Jessica was doing what she's known for and kept everything running smoothly.

When trying to convince me of the merits of San Francisco instead of San Jose, Dan from Weebly attempted to interject a point about dating being more prolific in San Francisco. His approach was a bit off. Dan tried to ask, "Do you have a girlfriend?" To which I replied, "No, I don't have a boyfriend." One additional round got the point across: "No, a girlfriend," he clarified to himself. A final recitation of "No, I don't have a boyfriend," cleared up any confusion. If you are a good Analytics person who likes SF and wants to work at Weebly, give Dan a call. He may even be able to hook you up with a girlfriend.

I monopolized Emmett's time for probably 30 minutes while having Sam Altman, Build Enginner Dude, Weebly Dan, and one or two others try to break into the conversation. Some break-ins were more successful than others.

I missed talking to Gnip Dude. He kept vanishing.
I missed talking to Paul S. I've a bone to pick with him.
I forgot to talk to the DailyBooth guys.
I'm not interested in moving to San Francisco, which writes off probably 90% of the startups in attendance.

Conclusions
Was it useful for the startups giving presentations? It's difficult to say. We'll have to wait a month or two until startups can report back how much usable talent appeared due to the event.

Was it useful for the attendees? Yes. An emphatic Yes. It's empowering to sit through stories of early-stage (and some not-so-early-stage), on-track startups. Thoughts of running a startup flow through so many of us, but not nearly enough people act on it. How about an Express-YC for more experimental ideas not needing a 30 question application?

Was it the right format? As I told someone before the event, and as Paul Graham reiterated during his opening talk: What matters are the people with whom you work. Go with amazing people and you'll be fine. A format of presenting the company's projects may be misguided. We should be looking for people to join. Have every employee in attendance speak for 30 seconds at the front of the room. Yes, some people are horrible with public speaking, but that's a data point in and of itself. Let's get the people to present themselves followed by one 20 second overview of their products.

Final conclusion? I need to get off my ass and get my own startup in gear. Time to redefine some paradigms.


-Matt
650-450-8826]]>
Fri, 18 Jun 2010 14:44:16 GMThttp://matt.io/startup/Work_at_a_Startup_Review/ud
introducing slippycomicshttp://matt.io/slippycomics/introducing_slippycomics/ucintro
I hide books from myself. The more dangerous the book, the more cleverly it gets buried. Recently I uncovered "xkcd: volume 0" again and lost an hour of my life re-reading everything on the spot.

It dawned on me: there must be a way to extend the productivity killing capacity of fluid comics to a mass audience.

This is around the time an HTML5 presentation was making the rounds. A slippy HTML5 presentation. A fluid HTML5 presentation. "Let's replace the damn slides with comics," screamed the productivity devil on my shoulder.

A few weeks later, we have slippycomics.

creation
slippycomics is a re-written version of the HTML5 presentation above. Most of the CSS is stolen, but the javascript has been completely rewritten.

Since the xkcd book inspired slippycomics, it was only natural to include all of Randy's comics. Next up are two comics which I always mean to read but constantly forget about: bunny and smbc.

All comics, alt text, and alt comics were scraped from their respective sites. The comics and alt comics for slippycomics are hosted on the Akamai CDN[1] as to not overwhelm the originating sites. In no way am I trying to steal or claim these comics. Each comic has a link back to the original site along with a link to the store for each comic every few frames.

[1]: If the CDN gets too expensive, I'll host directly from my servers.


tech stuff
front end site: html5, css3, coffeescript (-> javascript -> yuicompressor), jquery (1.4.2 + jquery.history + jquery.cookie -> closure compiler)
back end site: nginx -> yaws -> erlang -> redis (via my in-progress redis client er); akamai -> nginx -> disk
back end scraping: python with urllib2 and lxml.html for finding comic URLs, titles, and alt text/comics. python stores everything using redis-py. python downloader for comic graphics after comic data is stored in redis.


controls
Up. Down. Left. Right. r. Spacebar. [numbers then enter]. w.a.s.d. Click on stuff. Mousewheel. Touch sliding left/right for iPad/iPhone. Forward/back history. Remembers your last position if you close your window and come back later.


future
If you want your comic added, email me. I can add as many comics as people would like. Accounts may even happen one day for personalization.

It will be trivial to make an automated comic submission process if enough authors want their content here and don't want me scraping their sites.

If you want your comic removed, let me know and you'll be removed with extreme prejudice and a frowny face.

There are some UI bugs. The backend needs to be able to accept automated updates from authors directly. The iPad/iPhone interface needs fleshing out.


contact
questions? praise? love? hate? hugs? rage? Direct it all at matt or at @mattsta.

Go forth and slip: http://slippycomics.com/

]]>
Fri, 11 Jun 2010 13:45:50 GMThttp://matt.io/slippycomics/introducing_slippycomics/uc
abouthttp://matt.io/slippycomics/about/ubhttp://slippycomics.com/]]>Fri, 11 Jun 2010 13:28:14 GMThttp://matt.io/slippycomics/about/ubGoing Nowherehttp://matt.io/found_things/Going_Nowhere/u9
Shankill Butchers




Nick has a great voice but a less-great audio recording setup. Still worth a listen or two or twelve.

Mad World


On the Bus Mall



]]>
Mon, 05 Apr 2010 11:44:53 GMThttp://matt.io/found_things/Going_Nowhere/u9
french love lost at seahttp://matt.io/found_things/french_love_lost_at_sea/u8]]>Sat, 13 Feb 2010 11:43:30 GMThttp://matt.io/found_things/french_love_lost_at_sea/u8Wadeinghttp://matt.io/found_things/Wadeing/u7

]]>
Wed, 20 Jan 2010 10:09:19 GMThttp://matt.io/found_things/Wadeing/u7
Apple January 2010 Event Keynote Leakhttp://matt.io/iworkforapple/Apple_January_2010_Event_Keynote_Leak/u6background
I've spent the last month in isolation meditating on The Device Who Comes from Apple. Based on sensory deprivation induced hallucinations and stalking Apple coworkers, I've come up with this outline for The Device Who Comes. Also, it helps I found a leaked draft copy of the January event script.

The leaked script had the actual product name redacted, so I call it i[REDACTED] throughout the script. The script below leaves out details relying on actually seeing the device (demo parts of the script), industry recaps, and the scripts of guests invited on-stage.

summary
The device will not be called iSlate. What is a 'slate?' It reminds me of a misshapen piece of granite chipped out of the earth with shards flying everywhere.

There is a good chance The Device Who Comes will not be prefixed with i.

The Device Who Comes will use a Natal-like control interface thereby pissing off Microsoft to no end.

Basic Description: 13" 300 ppi multi-touch screen with free-space gesture recognition. "Giant iPhone" mode (for App Store apps) and full OS X mode. First client device to fully rely on Apple's new personal content hosting service. Perfectly designed reader functionality supporting books from iTunes (they may rename iTunes at this point) and supporting all open data formats for books (pdf, mobi, epub). Also, a new custom Apple eMedia format will be released for richer content consumption experiences.

Wireless Partner: Verizon Wireless.
Initial Content Partners: NYT and Conde Nast.

Prices: $1199 with free 2-year Verizon Wireless access. $699 with $30/month two-year Verizon Wireless contract. $899 with no cellular radio included (wifi/bluetooth only).

draft keynote
Thank you for coming. We're going to make some history together again today. Welcome to the future.

[industry specific recap]

This is a day I've been looking forward to for six years. Every once in a while, a revolutionary product comes along that changes everything. And Apple has been -- well, first of all, one's very fortunate if you get to work on just one of these in your career. Apple's been very fortunate. It's been able to introduce a few of these into the world. In 2007, we introduced the first iPhone, and it didn't just change the way we all use phones, it changed the entire mobile landscape. Well, today we're introducing three more revolutionary products of this class. The first one is a widescreen electronic reader with a full color, highest quality, multitouch screen. The second is the most portable Mac ever developed. And the third is a breakthrough service for managing your digital life. Electronic reader, extremely portable Mac, managing your digital life. Reader, portable Mac, digital life. Reader, Mac, Life... are you getting it? These are not three seprate devices. This is one device. And we are calling it i[REDACTED]. Today, Apple is going to reinvent the tablet computer, and here it is.

So, before we get into it, let me talk about a category of things. Current eBook readers on the market use something called eInk displays. These displays require no power while showing a page, but they only work in black and white and they are slow to turn pages. Slow to turn pages. In a book. Why do people use these? Some eBook readers also have Internet access, except everything is black and white and slow to change pages. Did you see tablets and readers from CES two weeks ago? There must have been over 100 tablet PCs and three dozen eBook reader models released. The "tablets" are all tiny and they all run Windows without regarding what form-factor the devices use. Desktop? Windows. Tablet? Same Windows. Same hair-tearing experience. The eBook readers all use the same underlying technology. Their only difference is what plastic case manufacturers use. Well, we don't want to just re-brand existing technology. What we want to do is make a leapfrog product that is way smarter than any mobile computer has ever been, and super-easy to use. This is what i[REDACTED] is. Okay?

So, we're going to reinvent the mobile computer. Now, we're going to start with a revolutionary user interface. It is the result of years of research and development. Now, why do we need a revolutionary user interface? Other tablets out there use a stylus -- a stylus -- or half-broken multitouch implementations. We can do better. i[REDACTED] is a full multitouch device, but it also has a camera built in. i[REDACTED] will also respond to just motions of your hand. See the camera there at the top? It constantly watches for gestures to be performed. In Reader Mode and want to go to the next page? Gesture from right to left and Boom. There's your page. Want to go back? Gesture from left to right. Want to scroll? Gesture in the direction you want to scroll. Boom. Isn't that amazing?

Let's talk about design. We've designed something wonderful for you to hold, just wonderful. This is what it looks like. It's got a thirteen inch screen on it. It's really big. And, it's not only the highest resolution screen we've ever shipped, it's also the highest resolution screen of its size available anywhere. It's 300 pixels per inch. Highest available anywhere. It's gorgeous. And on the front, there's only one button. It's the home button everybody is faimilar with from iPhone. Let's take a loook at the side. It's really thin. It's thinner than any tablet out there, at 8 mm. Thinner than an iPod touch. It's really nice. And we've got some touch spaces on the sides for quick navigation, along with volume controls. Let's look at the front. We've got a five megapixel camera and microphone built in to the top. Video capable. We've got a headset jack for listening to music. We've got a quick push-to-sleep button just like iPhone. And on the bottom we've got our 30-pin dock connector for charging.

Now, we've also got some stuff you can't see. We've got an extra-long-lasting battery. 8 hours of full use and up to 30 hours during light use. We've got an accelerometer so we can automatically switch from portrait to landscape. We've got an ambient light sensor for automatic brightness adjustment. We've got wifi and bluetooth built in. You can pair i[REDACTED] with your iPhone to see incoming calls and send text messages directly from the i[REDACTED] through iPhone.

So, now, let's take a look at Reader as part of i[REDACTED]. We've never offered books, magazines, or news for sale before. We're proud to announce a partnership with Conde Nast and the New York Times who will be creating content exclusively for i[REDACTED]. What other electronic reader gives you an insanely great mobile reading experience with full color videos and all the benefits of the web built right in? None that we've found. How do you get real-time, live updating content on i[REDACTED] no matter where you are? We'll get to that in a minute, but first let's show you hour Reader works. [DEMO]

Isn't that great? Now how do you get real-time articles on i[READER]? We've partnered with Verizon in the US and our iPhone partners internationally. You have real Internet access anywhere with i[READER]. [DEMO]

All right, now I want to show you something incredible. I want to show you how you get all your music, videos, books, photos, documents, email and more on to i[REDACTED]. This is a new way to manage your digital life. Over the past few years we've spent over $800M building data centers. $800M buys a lot of hard drives. When you have a desktop, laptop, iPhone, and i[REDACTED] we wanted to make it as simple as possible to keep everything up to date. Your home computer has an always-on internet connection. iPhone has an Internet connection anywhere. i[REDACTED] has an Internet connection anywhere. See a pattern? We're going to store everything for you. We'll stream music, stream videos, synchronize yours last read book positions, sync photos, documets, email, and everything else. On i[REDACTED], you have access to your entire library without having to download anything. Let's see how it works. [DEMO]

Let's talk about apps. On OS X, anything goes. On iPhone OS, we approve every application to maintain the integrity of the network and OS. You can't have people destroing their phones by downloading a malicious application. People live and die by their phones. But i[REDACTED] is something new. Do we open it up completely or do we approve everything? We've thought long and hard about the pros and cons each way. We decided: we'll let you choose. i[REDACTED] can run a full desktop OS X or i[REDACTED] can run in an iPhone-like mode where it only runs App Store apps. All i[REDACTED]s are in "giant iPhone mode" by default. If you go into the configuration menu, you can enable full OS X. We've satisfied the needs of every day users and advanced users alike. Simple mode gives you a giant, visually stunning, iPhone-type interface. Advanced mode gives you full desktop OS X in its most portable form ever.

So what's under the hood? i[REDACTED] has a super powerful dual-core 2GHz ARM CPU developed by Apple along with 2 GB of RAM, 802.11n wifi, and a cellular Internet connection. We talked about our plans for an always-on streaming device with all the US mobile carriers. Verizon is the best suited to meet the needs of our users now and in the future. They are building out the fastest nation wide data network and we've negotiated unlimited Apple streaming access on i[REDACTED]. All other Internet use through i[REDACTED] is capped at 10GB of celluar data per month. Data anywhere. Data everywhere. The bulit in five megapixel camera can be used for video chat anywhere. We have a gorgeous 13 inch 300 pixel per inch screen developed specifically for i[REDACTED]. It shows text sharper than a laser printer and images sharper than high def.

Battery life. Tablets out there have pretty low battery lives. We've managed to get eight hours of battery, and that's for watching videos and a lot of Internet browsing. That's eight hours of streaming video and audio. With light usage of just Reader and browsing a few websites, the battery life goes up to 30 hours.

So what should we price it at? Well, what do these things normally cost? First, what does i[REDACTED] replace? It can replace a MacBook you carry around when you don't really need the full power of a MacBook. That's at least $999. It replaces an ebook reader. A comprable ebook reader costs $500. We're including two years complete access to all Conde Nast and NYT content -- that's $1200. We're also including two years of our revolutionary media and document streaming service. i[REDACTED] replaces almost $2700 in other devices and services. What should i[REDACTED] cost? We've come up with three configurations. The first includes two years of Verizon Wireless unlimited data transfer between your streaming media and i[REDACTED]. i[REDACTED] + always on Verizon Wireless = $1199. The second lets you add i[REDACTED] to a Verizon Wireless account for $30/month. Two year contract required with Verizon Wireless. $699. The third option is i[REDACTED] with no cellular connection -- wifi only network access. That's $899. Now, when's it going to be available? We're going to be shipping these in May. We're announcing it today because with products like this we've got to go ahead and get FCC approval which takes a few months, and we thought it would be better if we introduced this rather than ask the FCC to introduce it for us. So here we are, and we're going to be shipping it in May in the U.S. We're going to Europe by the fourth calendar quarter of this year. Then Asia after that.

The Mac in 1984 is an experience that those of us that were there will never forget. And I don't think the world will forget it either. The iPod in 2001 changed everything about music. The iPhone in 2007 changed everything about mobile communications, and we're going to do it again with i[REDACTED] in 2010. We're very excited about this. There's an old Wayne Gretzky quote that I love. "I state to where the puck is going to be, not where it has been." And we've always tried to do that at Apple. Since the very very beginning. And we always will. So thank you very very much for being a part of this.

Questions?]]>
Tue, 19 Jan 2010 08:24:21 GMThttp://matt.io/iworkforapple/Apple_January_2010_Event_Keynote_Leak/u6
abouthttp://matt.io/iworkforapple/about/u5
-Matt]]>
Tue, 19 Jan 2010 08:06:35 GMThttp://matt.io/iworkforapple/about/u5
Hazards of Workhttp://matt.io/startup/Hazards_of_Work/u2Day job. The eternal safety net. The side projects can wait -- nay, must wait -- when day job comes calling.

Day job. The infernal contraption taking inputs of time and effort resulting in output of electronic signals which trigger my bank account balance to rise ever so slightly twice monthly.

Day job. Disruptor of sleep. Left to my own devices, I fall into 26 hour days. 17 wakeful, alert, upbeat hours followed by the inevitable sleep of what dreams may come. Day job forces a wake-up time. Day job doesn't care if you can't sleep until 3am. You'll still wake up at 7am to start the next "day." They don't call it day job for nothing. Cyclical sleep deprivation is a constant when day job is around.

Day job. Killer of nutrition. We turn into scavengers seemingly never knowing where our next meal comes from. "Better eat a big meal, I might be working late," the inner voice prods. Left to my own devices, I eat every few hours. Day job alters that to eating 1.5 times between sleep (let's say 0000) and dinner (let's say 1800). 19 hours. 1.5 eatings. No wonder the bodies don't know how to regulate their input.

Day job. Breaking retention time. Side projects. Soon-to-be-Startups. The complex brain state can't be maintained part-time. It's relegated to spans of free time. Weekends. Holidays. Weeks off. After days back at day job it's as if the side projects and soon-to-be-Startups don't exist. They've fallen out of your internal fifo queues. Projects with a life of weeks take months. Projects with a life of months take years.

Day job. Getting home. Tired. Angry. Annoyed. Hungry. Oh, look. There's fresh tv shows to download. That'll get my mind off the absurdities. The shows comfort. Visual drugs let us forget our dreams. The dopamine loop kicks in. Another show is played. Looping. Another show. Loop. And yet another. Are we tired yet? Just one more. One more. Almost tired now.

Day job. Starts over again. 5/7ths of the week. It starts over again. The pain is forgotten. Numbness is natural, dreams are undesirable, and failing is fantastic.

Day job. Comfort. Warmth. Security. Income. Average. Boring. Simple.

Day job must fall. Startup must rise.

]]>
Mon, 04 Jan 2010 15:21:52 GMThttp://matt.io/startup/Hazards_of_Work/u2
how make living playing musichttp://matt.io/found_things/how_make_living_playing_music/tzhttp://www.dannybarnes.com/blog/how-make-living-playing-music but I found the formatting there so horrible I couldn't read it. Here it is with a touch of Text::Autoformat]


I hear so much complaining about this subject, I just wanted to lay my practical experience on you. Free.

First, three pre-conditions:

1. If you are a very materialistic person, skip this article, I don't think you are going to like what it says.
2. If you don't have the music where you want it art-wise, you might want to go work on that, this article isn't going to help you much either. You will be better off by practicing and studying and working on your music instead. You will need to get the art pretty close to where you want it, before you should worry about making much of a living out of it.
3. Determine if you are actually called to be a musician. If you aren't called, all the gyrations in the world, won't make it work. If you are called, no matter what you do, it's going to work. This determination will solve most of the problems you are going to encounter.

Assuming these three conditions are met, you are financially workable and you have the music where you want it and you are surely called into the art, here goes, in no particular order as I am want:

a. Keep your expenses very low. Read that one again. Move someplace cheap. Drive a good used car. Do all the things it takes to be a secure un-monied person. You have to have health insurance. You have to have a reliable car [unless you live in nyc or something]. You have to have some money in savings. You have to pay your taxes. Don't have a big expense of alcohol or drugs or any drag on your system like that. I wouldn't even smoke. Use your head. Spend very little, save as much as you can and don't get into any big expenditure until you can afford it, maybe never. Buy your gear used. Research as much as you can. Think about it really hard before you part with a dollar. Learn how to honestly add and subtract without emotion. If you spend more than you take in, you lost money. I can't tell you how many folks that I run into that have trouble with this. If you bring in more that went out guess what? You just made money. Stick to this low-overhead model, if you end up making a bunch of dough, you already know how to deal with it. If not, you still get to keep working because you don't have a bunch of stuff that you have to dust and pay for. The more overhead you tack on, the harder it's going to be. And the easier it is to get knocked off course.

b. However, don't be a cheapskate. Tithe or donate faithfully whatever your heart tells you to do. Pay your band as much as you can. Never withhold a laborers wages. Tip well. Give street musicians money. Become involved in charity work.

c. Be totally square on your taxes. Render unto caesar that which is caesar'S. If you try to fudge on this, it will come back to bite you every time. Get receipts for everything, 1099 everyone no matter what, unless they are a corporation. Be totally on top of this or you are burning money in a pile on the lawn. Claim every dollar you make and take every deduction. Otherwise you are a drag on the system. Keep perfect records.

d. Your basic infrastructure will have to consist of these things: a good lawyer, bookkeeper, cpa, doctor, a mechanic, an instrument repair person, web person, and someone in your circle that will always tell you the truth. Maybe a backup of each one. And do what they say. These are all musts, even for solo acts. Then later you can add a good agent. Then maybe a manager if you have lots of stuff to deal with like a label. You can grow from there. If you don't assemble a good team of the first eight people on that list, you are likely to have problems every time you turn around and you might not have a way to fix them.

e. If you are going into a deal with any entity, seek two things:
1. The arrangement must be win/win. Win/lose is ultimately lose/lose. Avoid that.
2. Make an agreement that either one of you can walk away at any time and everything is cool.

f. Keep working on your art. Keep taking lessons and studying and working. This is the main art strategy. Research, learn, study, experiment, develop, edit.

g. Don't be afraid to do other things to make money in the short term. This can be a very rewarding experience. Historically musicians have been barbers and bartenders and all kinds of stuff to make ends meet. This is totally fine. Don't worry about it. It's cool. Do what you need to do. Waiting tables will give you lots of stuff to write songs about. I used to call myself the king of the part time job, because I could get up out of my chair at any time and go get a job of some sort. Not that it would be the greatest job of course, but I could go and get something going. I've cleaned pools, painted apartments, done maintenance work, taught music, worked in a factory, threw newspapers, drove a delivery truck, cooked, all kinds of stuff, and none of it killed me. Through it all I was able to keep practicing and writing music and studying what I was doing. Bills? Hey no problem, go flip a few burgers and I can pay that and get back to playing the banjo. Get a job in a dance band whatever I have to do. Just live within your means and you can avoid so many hassles. Hassles interrupt your practice routine.

h. Keep your art the main focus. It isn't about you it's about your art. Do what's good for your art and don't draw attention to yourself as much as the art. If your main focus is on the art, waiting tables is no big deal because you are doing it to support your art. If your main focus is you, you are not going to like waiting tables. You will feel like you are way too good for that.

i. Avoid the performance mentality. I know this sounds ridiculous in a performance based industry. But think about this. Here is a recipe for disaster. My value = my performance + other people's opinions the reason why, is that someday, you are going to have an off day and/or someone is going to criticize you. If you put your value in the world like that, you are going to have a bad time of it. I speak from experience. I only learned this at the age of 46. finding my true value fixed this for me. [Write me if you want to know what it is.] But establish your value outside of how well you did on the gig and what the papers said about you. Otherwise you are going to be miserable and you are going to make everyone else miserable. Somedays you play better than others. This doesn't make you a great person. Somedays you make lots of errors, this doesn't make you a bad person.

j. Don't gossip. Gossip means you aren't in the problem or the solution, you are just talking about someone and probably gaining pleasure from something bad not happening to you or envying something good that happened to someone else. Spend your energy on getting better at your art.

k. Record labels. They can help or they can drag you down. Here's the scoop. If they expect you to be the primary distributor of the product, don't sign the deal. The typical deal is a 90/10 split, you get the ten minus every expense related to the project. Thus you are paying for everything and giving the label 90 percent of the gross. Read that sentence again. If they aren't really really offering you something good in terms of promotion, or something....some tangible quantitized tie-in to something bigger, skip it. You can hire that stuff yourself easier. Talk to other artists on the roster and ask them what they think. Any more, if you are an emerging artist, it's going to be hard to find a label home. They are losing so much dough they only want for sure money makers or somewhat less money losers on the roster, and they are dropping folks right and left. This is all good for you. Take heart. It's a 90/10 deal and you get the 10 and they want you to be the primary distributor of the product plus pay for the whole deal, those are not very good terms. In addition they will charge you eight bucks plus shipping for your own cds that you can make for either zero or one dollar. And they might complain about every little detail. Again if they really have an idea for a bang up thing they are thinking of, by all means have a go. If they are motivated and have a track record and have ideas and are workable, they can really help. However, you might want to have an out. Have an out clause in there. Shooting from the hip, i'd tell you to avoid the whole thing and do it yourself.
It's very likely that the person that brings your act into the label fold will get fired. Then you can get stuck with four years left on the deal and no one will return your calls. Then they just hope you will get another deal and someone will buy out the rest of the contract. Lot of bands close up shop at this point. There are some labels that operate with different models. I have had very good success with them. They tend to be more punk rock style outfits. You might want to investigate that. The standard deal referred to in the preceeding paragraph is pretty hard to profit from unless the contract is on your letterhead. The punk rock deal goes something like this, all the black ink goes in a list, all the red ink goes in a list, find the difference, split what's left if it's a positive number. Fifty fifty. These are really the only deals I ever made money on. The point is, there are some other ways to look at stuff contractually. If the deal is win/win, great. If it's win/lose, skip it. If the label in question is locked into doing contractual things a certain way, this won't be for your benefit. You are creative, your business arrangements can be creative.

l. The main business strategy is to build your own audience. If you have a draw, agents, labels or investors [which I do not recommend] and stuff will come to you. If you skip this step and start trying to talk to industry people and you don't have a draw yet, you are going to be sorry [unless you are really hot looking or have a famous parent and/or willing to sign away the rights to the whole thing of course]. Build your own audience. If you can sell your own records that you make yourself and do your own shows, you can attract the attention of industry folks and get your calls returned. Then you probably won't need them unless you want them. That's a better bargaining position for you. Work on your draw. If you don't have a draw, these are some likely things to look at: where you are playing isn't the right place the music isn't there yet the time isn't right

In any case, the answer is to forge ahead. Keep doing it. Always keep writing and practicing.

Keep working on finding more and better places to play. And new contexts within which to place your work. If something feels right, it probably is right. If you are having to bang your head against the wall in regard to something, it may be better to drop it sooner. The longer you work on something that isn't going to work out job-wise, I think the more time we waste.

I wouldn't get too hung up about opening slots. They are okay and you can increase your draw, but as far as that being the principle strategy you are using, it may not work. The old model of thinking that if you open for someone and do a good job you can get some of their audience interested in your work is not really that reliable. Find a new model. If you meet someone who wants to work on your team, and you are thinking of hiring them and they offer this as the main strategy, this is not a creative workable person. They are working on business models that are decades old. This ploy will work sometimes, but it should be part of an overall deal, not the main thing. Just like if you went to interview a financial advisor and he said, "what we are going to try to do is to buy low and sell high," and behaves as though he has just isolated the plutonium isotope. You might need a little more horsepower upstairs than that if you get my drift.

I work for free when it's kind of my idea to do so. If someone else suggests it, I tend to pass. I also pass on a job where they say they aren't going to pay you but you'll sell lots of cds. And when I did not adhere to this, I was sorry.

I'm not really a self promotion person, and find that sort of distasteful. In my experience the strong self promotion vibe alienates people or attracts folks that you don't want to work with. Maybe I just didn't do it right but this did not work for me. I've had much better results endeavoring to let the art speak for itself.

m. Don't expect to get paid more than you can bring in. If you draw ten people, and the cover is ten bucks a head, you gross one hundred dollars. Not five hundred. Don't get mad at the agent, club owner or whatever because of simple math. You drew ten folks. Guess what? That's better than nine. If you want a raise, figure out how to draw more folks. This is not as mysterious as some would suggest. But you can't ask for more than you bring in the door. If you don't believe this, try producing some concerts of your own.

n. You may not want to hire sidemen that get too worked up about money, it can be hard to make these folks happy. Also when it comes to hiring musicians, you may have to live with them at arm's length for a long time and be involved with them about emotional issues like money and life problems and stuff. You may want a person that's easy to get along with even if they are a little less sharp musically. Of course getting both is best, but if you have to take one or the other, take the one you get along with a little better. If you are in a place where you don't have a lot of choice, you may be forced into hiring someone that's tough to be around. Replace them when you can. Really the best players I know are also the nicest folks. Except for one or two. Many times, in that world of musicians that are struggling to make a living, but haven't really gotten there yet with the music or with the people skills or what have you, they will be the most difficult to deal with. They over-compensate by talking too much, or acting like they know everything, or showing up drunk or being really critical or whatever. When folks have it together, they are at ease and play great, and know when to lay out and stuff. They are also more expensive. It's totally fine and many times necessary to use different players on the recordings than in the shows. If you are a leader, do this with no guilt. If you are a sideman, get ready for it and don't complain. It has to be this way. If you don't believe it, trying putting out your own record. You'll soon see why when you go to record. Sidemen, you can
always practice and take lessons and get your tuning and timing together. Leaders again, get their tax id and report every dollar that transacts. If someone is upset about this, you can't use them. Period. Never fudge on taxes.

o. You really won't be able to work that much in the town where you live. And there will probably be a morass of musicians in your hometown that aren't really committed to the lifestyle that haven't really developed their art that will be complaining loudly about how hard it is to make a living and whatnot and you can easily get sucked into their trip. You'll be better off traveling to various places and developing that. Use local shows to try out new stuff, play with different folks, have fun, play for the home town crowd, etc. but typically you won't be able to work that often at home. Maybe twice a year or something. Don't worry about that. Your market is the whole world, not your hometown. Negativity is a sign to alter the course.

p. Don't let anyone tell you that you can't make money playing music. Six of my pretty good musician friends are millionaires. Three of them multi. Three of them play music that most folks would surely comment, "you can't make any money playing that." Don't tell those guys. Five of them are the nicest people you would ever want to meet. One of them is as mean as a snake. There you go.

q. I would suggest being able to do different things. If you write songs, maybe you can sing on other folk's demos. Maybe play guitar in someone else's band. For years I taught music lessons in a music store. Many folks I work with have a little studio and also play in someone's band. Or they are a chef or tax person on the side. This is all very healthy. I know several folks that are sidemen but have their own writing deal or what have you. This is a good course to take. That way you can take a hit and keep moving. The world doesn't grind to a halt because your label went under.

r. Be wary of someone that talks about gear a lot. Also be wary of folks that tell you how great they are. Stay away from complainers and folks that don't have their lives somewhat together. Sometimes folks need some ministering, which is certainly what we are called to do, but if you take someone out on the road with a big jones, you are going to be sorry... Or otherwise get involved financially, look out. Don't make your own problems or agree to be in a messed up deal. Drama is always bad. Never make a financial agreement with someone that has no problem getting paid for not working.

s. All the trouble in the world is going to come for you in two ways. The things you say, and the things you agree to do. Be very careful about these items.

t. Build alliances. Let's say you play some weird kind of music, make contact with someone in another city that does something similar and offer to set up a concert for them in your town. Maybe they will later help you to play their city or something. Work it out with them. If you can't get into a particular festival, why not have your own festival? Get some like minded bands together, the venues would love to turn over the night to you to produce your own gig, and do it yourself. Sometimes you can do stuff like that yourself easier than you can talk someone else into doing it for you and then paying you, think about that. Going to that big music conference is out of the question for some reason? Why not have your own conference? It might be cheaper to fly the guy in you are wanting to have see your band. That way you only have to put one guy up, rather than having the expense of flying a six piece band to los angeles and have one guy come out out to the show that lives there. He may blow you off anyway. It would probably be cheaper to fly in six a and r guys to where you are and put them up and have them come to the show, than it would be to take the band out to them because of the gear and salary. You also could have their undivided attention, within reason. Don't keep saying "well if I had a label or agent or manager, then I could be happy." Forget that. Forge ahead with your music. Keep working. Develop the music. Come up with different ways to do an end run around conventional wisdom. If you are really called to be in music, the right people will present themselves at
the right time. Build those alliances of simpatico musicians, writers, studio guys, label guys, radio guys. Be nice and help others. I have been fortunate enough to be close friends with lots of folks that are way better at music than I am. I take constant inspiration and encouragement from these folks. I think this has been really good for my work.

u. If for one second you think you aren't getting the recognition your talent deserves, banish this thought immediately. If others tell you this, ignore it. Just keep working on the music. You are probably right where you are supposed to be, learning and doing what you are supposed to be learning and doing.

v. If there's no social context for the music you are making, don't be mad if no one comes to the shows or buys the music. Or if only very few people do. In that case the reward has to be the music. Hey that's a great deal. Also you have lots of freedom to do different stuff. There's no one to alienate. Let's face it, sometimes having no one at the show is a great indicator that you are onto something. I'm serious.

w. Robert keen told me he never regretted firing anybody that he fired, and I agree. If someone is a problem, and they won't fix it, get rid of them. It's okay. You both will be happier.

x. Don't waste materials and time giving a cd to someone unless you are fairly sure they will actually listen.

y. Avoid folks that make your job harder. Sometimes people gum up the works, even when they have a smile on their face. You'll get more done the less of this type you deal with. When you ask someone a direct question and they go into a convoluted story about something else, get ready for the hassle.

z. We are all blessed with different talents. This is as it should be. Don't be upset with someone that doesn't have your talent for something, and don't feel bad because someone else got some talent that you think you want. Move towards grace.

z. I have a system, where if I sense that the gig is going to get weird before I even get there, I cancel the show and walk away. In my experience, if something goes awry before you even get there, it won't magically get better if you commit a bunch of dollars and time towards it. Because of this, I can't remember the last bad gig i've had. Example, let's say i've booked a show next year with a person that I don't really know that well. And as time goes by, he keeps wanting to chisel away at our arrangement, or add stuff for me to do, or whine or complain about the situation, I would cancel the show. Time and time again I learned that it only gets weirder and more difficult when you get there. This is better for the buyer too because then he or she doesn't have to worry about my show anymore. If the buyer isn't really into it, or at least somewhat into it, seriously consider passing on the show.

z. Have interests outside of your art. Especially if you can do this on a non-performance basis, where you can just enjoy the activity and not analyze it to death and be real critical of your own work and stuff. It's so easy to burn out if you do one overwhelming thing for about twenty or thirty years. Sometimes, I just don't play at all and don't think about work and mess around with my sailboat, or work in the yard, or something. Ride the motorcycle. Giving myself a break from the pass/fail mentality. I like just being a regular person.

z. Think of your art as a work in progress. That takes the heat off of it having to be perfect all the time. Keep working on your art, your vision, your catalog. Dedicate your work life to that, and things will work out.

Okay i'm out of letters.

[These are all just ideas, and you may have your own way. Good. Also these are different components and you have to make a sort of stew of them. Maybe you have a little more of one, use a little less of the other or whatever it takes to make it come out right. I've made all these mistakes myself in the trial and error process, which is a fine way of doing things except for the error part.]

Barnes
(from http://www.dannybarnes.com/blog/how-make-living-playing-music)]]>
Fri, 01 Jan 2010 13:31:56 GMThttp://matt.io/found_things/how_make_living_playing_music/tz