Rebecca Murphey

Eng ladders, promotions & glue work

2021-12-23T00:00:00Z

(Many disclaimers belong here, including, but not limited to: this probably applies differently or not at all in organizations without formal ladders and review processes; most and perhaps all of the early-career women I’ve coached on this have been white, and many but not all have been from non-traditional backgrounds; it is OK if you don’t want to climb a ladder; some companies are evil, some managers are bad, and also the patriarchy; “the business,” at the end of the day, is nameless faceless capitalism; and finally, this may read as harsh in parts but I’ve found that many people appreciate and benefit from the directness.)

A lot of the advice I might share from my own early career, such as “make friends on IRC who will work at cool places in 15 years and refer you” or “participate actively in the comments section of your personal WordPress blog, where you write about jQuery” just doesn’t feel very applicable these days.

There’s one piece of advice I find myself giving over and over, influenced by my more recent years as an engineering manager. This comes up all the time when I’m talking to early-career women pursuing a career and professional growth in software engineering — that is, women who have ambitions to climb the engineering “ladder.”

A gross oversimplification of such a ladder might progress something like this:

You write code to complete tasks that are valuable to your team. You have relatively few opportunities to make big mistakes, but when you do take advantage of those opportunities, you learn from it. You’re still figuring out exactly when to ask for help.
You complete tasks in order to help your team complete projects. You’re good at unblocking yourself. You still make mistakes — you will always make mistakes — but some things that used to feel hard or scary are easier. You’re developing a solid understanding of the systems your team works with and depends upon.
You own and drive small to medium projects and contribute to defining and achieving your team’s goals. You’re good at unblocking others on your team, and others on the team look to you for guidance. You have a growing sense of the architectural world outside your team.
You own and drive large projects. You guide your team in defining goals that align with the org's mission and vision, and you contribute to defining your org’s goals. When you’re at your computer, you’re more apt to be reading or writing docs than you are to be writing code. You have a high-level understanding of multiple business-critical systems, and are well-versed in at least one critical system.
You’re working with other leaders to set a mission and vision for your org that aligns with business needs, and then collaborating with those leaders to guide your org accordingly. Coding is pretty rare, but your technical instincts, experience, and understanding are essential. You have at least a high-level understanding of all critical systems and how they interact, and deep knowledge of several critical systems. You bring experience from other relevant companies.

Three things I’ve learned about this type of ladder: first, that people, process, and political skills (basically the sort of stuff that Tanya Reilly calls glue work) go from useful to valuable to essential as you move up the ladder; second, that these skills do very little to help you climb the first couple of rungs on the ladder; and third, that there are more than zero engineering managers who know this but don’t know how to communicate it to their reports who are struggling to get promoted for exactly this reason.

So, here’s the thing that I tell so many people who are trying to make those first couple of jumps (and maybe this will give managers some language they can use too):

When “the business” is considering whether to move you up the ladder — which will plausibly result in you making increasingly more money for the rest of your career — your manager has one fundamental question they need to answer:

Will I be able to review this person as “meeting expectations” at the next level during the next review season, in a way that is true to the letter and spirit of the ladder?

(Because of course any decision that wasn’t true to the letter and spirit might be characterized as favoritisim or bias.) A manager can’t “jk” about a promotion they end up regretting when they can’t defend it after the fact; often the only way to take it back is with the dreaded performance improvement plan.

Perhaps your manager has pitched the impact of your glue work to get you good performance ratings in the past. That value is at best tangential to the question of whether you can meet the business’s expectations of an engineer at the next level.

“But the business should value a wide variety of skills; every engineer shouldn’t have to be the same!” Well, the business does value a wide variety of skills ... in a wide variety of job titles, at a wide variety of prices. (Remember, “the business” is, at the end of the day, nameless faceless capitalism, as disclaimed above.) You are being evaluated against an engineering ladder in exchange for the engineering compensation, and that ladder wants you to write a lot of code at the lowest rungs.

Charitably, the ladder doesn’t value non-coding skills at lower rungs because those skills take time away from gaining engineering skills and experience that will be necessary to get to the next level. But no one is strongly incentivized to tell you that — your team likely values the non-coding skills you bring, and your manager especially appreciates them, because you’re making their job a lot easier. It might be up to you to figure it out.

Uncharitably, “the business” is getting glue work at a steep discount, like grocery stores getting you to bag your own groceries because they gave you control of the scanner thingy and the ability to interact with one less human (unless you are buying alcohol, but then at least the interaction is brief). Your skills and your willingness to accept the tasks mean that “the business” didn’t have to hire someone extra for that particular skillset, and early-career engineers are relatively inexpensive to begin with, relatively easy to get to a sufficiently productive steady state, and relatively easy to replace (in the U.S., at least) if they don’t have a clear trajectory. Did “the business” set out to be so zero-sum? Probably not. And yet.

If you’re still in your first couple-few years and you aren’t hands-on-keyboard writing and reading and reviewing code and talking about code most of the time you’re working, if you haven’t yet established that trajectory I talked about ... you are a good and valuable human! Maybe we could be friends! But the first conversation we’re going to have is how you’re only hurting yourself if you aren’t focused on the ladder if what you want is to climb the ladder^*. And if you’re realizing that you enjoy your not-coding time at work more than your coding time, we should talk about that instead. There are a lot more jobs in this industry than I ever imagined when I was getting started, and only a few of them require teaching computers to do things if it turns out that’s not your thing.

(A good work friend of mine who knows me quite well once had the guts to suggest that maybe I’d enjoy being a product manager. I was offended and he was right and I was offended more because he was right and I knew it. I wrote him a long note in the wee hours of the morning when I couldn’t sleep because our conversation had upset me so much. I told him how it felt to have a senior, white male engineering leader suggest that maybe I was a better fit for a product role than a role as a fellow eng leader, and how it felt like it went against everything I had worked hard to achieve to even CONSIDER the idea.)

(And then a few weeks later I took the job and I was very good at it. It made me a lot better at a lot of things, and gave me a break from things I needed a break from. I took the job I have today — as an engineering manager, again — precisely because of the product aspects it includes.)

If the ladder is what you’re after, success is going to depend on being open to — and soliciting! and incorporating! — good-faith feedback, even (especially) when it’s hard to hear. Insist that your manager provide direct and honest feedback, and focus your attention on feedback that is specific to you. Ask direct questions, like: “If you were me and you wanted to get promoted, what would you be doing more of? What would you be doing less of?” Peers and mentors can be great support when you’re trying to get leveled up, but they may have a limited understanding of their own strengths and weaknesses when it comes to the ladder, which can skew their perspective on “what it takes.” Again, ask for feedback that’s specific to you. “More of/less of” questions can help focus the feedback, and questions like “What's something I could have done better in the last month?” can give people the permission they need to say what you need to hear.

Finally, to managers: I know it’s not fun to have these conversations, especially when you’re having them with someone from an under-represented group of which you are not a part, especially when you really want to see that person succeed because otherwise you’re worried you might be part of the problem. I have avoided these conversations myself, and I have watched people avoid these conversations and not pushed them to do the right thing. You (and I) do no one any favors by extending a protective wing that shields them only as long as you’re their manager. I have seen direct, difficult feedback on this topic change the course of people’s careers for the better — often ending with the person thriving as an engineer who finally understands how to grow and thrive.

Maybe it’s useful to shift your mindset from “I want to see them succeed” to “I want them to enjoy success.” The first takes an implicitly narrow view of success: unstated is the rest of the sentence “... at what they’re trying to do.” The second focuses on a success that’s defined primarily by their enjoyment of it. Their enjoyment might actually be in climbing an engineering ladder, even when they’re a bit miserable in the moment — and they need you to help remind them of that and help them stay focused on that goal. Or their enjoyment might be in seeing ideas turn into reality on a screen — and there are a whole, whole lot of ways to work on that. Be the one with the guts to help them find their way to enjoying success, whether “engineer” is in the title or not.

* Hope for the Flowers is a cherished book from my childhood that I now see is categorized in “Christian Books & Bibles > Christian Living” and I’m not sure what to do with this new information, but I still think it’s worth owning in its paperback form.

all too well.

2021-12-22T00:00:00Z

You landed in Tokyo for work in January 2020 and there was a text from your mom saying your brother was in the hospital with a surprisingly bad case of pneumonia. You’d seen him just a few weeks ago when the whole family all got together for Christmas, which was exhausting. You and your partner agreed on the trip home that you would skip next year.

Your last night in Tokyo, outside the Naka-meguro station, you said goodbye to a friend who had just moved to Japan. You didn’t linger because you’d be back in just a couple of months.

There were people with masks on the flight home, but not too many more than normal, really, but you were definitely at least glad to be heading back to the U.S., far away from there, with a stop in Seattle before you headed home. Coworkers with family in China had cancelled trips home because of a virus. You listened, on the way back, to a podcast about the start of World War I.

Saturday afternoon drinks and board games with friends and their kid at a packed pub, the same weekend you went to the grocery store and bought toilet paper, pasta, and an unreasonable quantity of canned tomatoes.

Taking your kid to school, trying to explain that his life might change dramatically any day now, but struggling to explain how because you can’t quite comprehend it yourself. Coming home and reading about how people in China were getting creative to make meals of the food they had on hand when their lockdown began. You call the pharmacy to see about getting an extra supply of meds, and they can’t fathom why you are asking.

March 4, the day that work asked you — well, told you — not to come to the office for a little while, and you started your morning watching the VPN collapse. You cut the tip off your finger that night with your new fancy knife while making dinner.

March, April. You come to terms with obviously impending death. Breakfast tacos at home. Wondering what it would be like to die before your parents and without getting to say goodbye to your kid. Legos at standup. Calm, capable people at work freaking the fuck out because no one can tell them what’s going to happen.

Making bread.

At the Airbnb you stay at in May, your kid takes your parents on a Face Time tour, showing them around the unfamiliar house, he’s absolutely marveling at the different styles of trash cans because at least it’s something new compared to the monotony of the past couple of months. Fishing at a nearby lake, no boats to rent or bathrooms to use because you might die. Wide berths around a few strangers, but especially the ones forsaking masks outdoors.

Neighbors converse across the street. Your kid makes do with retirees as friends, joining them for “dog party” in front of the house. Summer afternoon on the screen porch, you explain to your kid that while, yes, the government does have a habit of using military-grade force against its own citizens in response to reasonable demands for justice and equality, we probably weren’t going to see the air force bombing our neighborhood anytime soon, and this among so many other things is what we call white privilege.

Convincing yourself that a 10-year friendship

(she hasn’t spoken to you since your attempt to come up with a shared solution for childcare in the fall of 2020 — back when you thought “if this wasn’t over soon ...” but also “at least we might have a new, competent president who will certainly do obvious things like send out free at-home tests and high-quality masks in his first month in office ...” — she hasn’t spoken to you since that plan didn’t work out and you probably could have handled it better than you did but everything was just a lot, then, and it was hard to fight for things anymore)

had maybe run its course and wasn’t in fact a gut-wrenching loss.

Weekend bike rides with your kid, but especially that time you got him to ride the whole 22 miles of the American Tobacco Trail and he was so proud. Campfires and marshmallows and bat-spotting in the back yard on a weeknight, just because. A whole Lego city on the dining room table, and suburbs in the living room.

TFG gets sick, and you don’t know yet that the stunning lack of consequences is just cruel foreshadowing. The scarcity of lessons you can teach your kid in that moment without lying is just as stunning. RBG dies and all you want to do is break things and scream. A year from now you’ll grasp at a blur of memories trying to remember the exact order of all the terrible things happening at once.

You live in a failed state and winter is coming.

The election, standing in the cold for hours with a volunteer for the Democratic party who just couldn’t bring herself to vote for the opponent to our state’s plainly racist Republican senator.

A new job, one laptop closes and another opens, no wistful moment walking out a door for the last time, and most everything else is the same. Christmas, at home, just the three of you.

The beach, just after New Year’s, your kid came with you for the first couple of days and it was a beautiful 70-degree day. Anxiously watching him play with other kids he met, wondering what the price would be for the 45 minutes of sheer joy you were allowing him in the bright sun and stiff ocean breeze.

Three days later, alone at the beach Airbnb, one then two then three separate TVs tuned to different cable news channels in a 1,000-square-foot condo, deciding that drinking alone at two in the afternoon was in fact the most prudent thing to do under the circumstances. Emerging from a drunken sleep in the middle of the night to watch the election get certified after all, and learning that we’d have a Democrat-lead Senate. Hope.

Taking the day off for the inauguration out of a profound sense of fear about what might happen, but what happened is you cried your eyes out when Amanda Gorman spoke, and then slowly realized that a young Black woman was the only person there who was capable of rising to the moment.

March. April. Work, vaccines, basking in the fever that says this is almost over. Buying a last-minute first class ticket across the country and feeling near-reverent about the banality of air travel. Discovering the joys of taking a camper to a state park 30 minutes down the road, just to get away.

Trying Prozac, just to see, and discovering — some what to your surprise? — that you have not, in fact, been OK.

A summer road trip, you and your kid, you stay at a campground for a couple of days and he plays with a dozen random kids on a jump pad that is basically a bounce house without walls, and he is full of joy. He’s old enough to not need you around so much. Hugging your parents again, visiting friends you haven’t seen in a decade, just because there was a time when it felt like maybe you couldn’t ever do that again. Your partner joins you halfway through the trip and takes the kid in her car for one leg of the drive home. He throws up after 15 minutes on a winding road. She is annoyed, of course, but you laugh about it together because this is by far the worst thing that has happened in a solid three weeks.

December 2021. Your kid is fully vaccinated. To celebrate, you go out for sushi at a perfectly adequate place. He thinks it’s the fanciest restaurant in the world.

Visiting an office, reverent in banality again. You write on a whiteboard and hand the marker to your coworker, who is standing in front of the white board next to you, in an office, without a mask. Simple acts simultaneously foreign and familiar. You have never met them in person before today. Your one-year anniversary at the company was last week.

The new president has been in office for almost a year. His press secretary, at a briefing, scoffs at the very idea that the richest and most powerful country on earth would send free at-home tests to every person in that country. She is not fired, and this explains a lot.

Your friend who lives in Japan is back in the U.S. for her first visit since the before-times. She tells you she’ll be in Austin three days from now and 20 minutes later you’ve booked a flight.

You rent a house together and stay up until all hours of the night and some of the morning ones too, talking about climate change and geopolitics and Japan and holding on to friends from your 20s, and also just a little bit about climate rebellion. One night we spent a solid five minutes just reiterating to each other exactly how amazing Taylor Swift is. None of it got old.

The weather in Austin is unseasonably warm. Forced childbirth is the law here in the state where you sit on the porch at midnight in December, but masks and vaccines are entirely optional. The senate is about to head home without doing anything about voting rights or social infrastructure, and you feel like you can write the script for the months between now and next November. You don’t even feel weird anymore about using the word fascist to describe the soon-to-be-ruling-again party.

You still live in a failed country that mostly doesn’t realize it yet, and you’re pretty sure no one is coming to save us, but you’re reminded, now, on this porch, that you’ll slowly get to start living in this mess with friends again, and that will make it a little more OK.

Omicron. Christmas. At home, again. Just the three of us.

Writing effectively in software engineering organizations

2021-01-23T00:00:00Z

My job as an engineering manager requires a lot of reading. Every day, my browser gains a few more tabs as I open internal documents that people share with me in the course of our conversations. In sharing those documents, people are seeking to provide me with more detail, background, and context about a topic we’re discussing. At the end of a day or week, there can be a whole lot of documents competing for my attention.

My job also requires a lot of writing. The written communication that my team and I generate is likewise an implicit request for the attention of future readers. Their attention, like my own, is a finite resource. Information overload is real.

If our communication isn’t crafted with purpose and intention, it places a burden on our audience: they’re left to decide whether the communication is worth their attention, whether it requires their feedback, what the key takeaways are, whether there are hidden action items or implications buried within, and so much more. They have to do work that we could have done for them, and that we’re best positioned to do well.

The culture of communicating via longform writing is strong at Stripe, and that’s led me to think a lot lately about what it means to be good at this. A lot of this is applicable beyond written communication, but written communication skills seem to be an area of … particular opportunity … for software engineers.

Important purposes, essential questions #

My thinking starts with considering the reasons we need to communicate in an engineering organization. Among them:

To influence decisions.
To summarize decisions.
To solicit input and drive alignment.
To articulate tradeoffs.
To share knowledge.
To share accomplishments or setbacks.

How do we effectively achieve these important objectives via written communication? Each of them requires a distinct approach, but they all require asking a few key questions:

Who is the audience for the communication? Are they busy? Are they familiar with the subject matter? Do they need to make or contribute to a decision about the topic? Does the topic impact one of their projects or goals?
What is the purpose of the communication? If we can’t complete the sentence “It’s important that <audience> read this document so that we can <purpose>”, then the document may be for recording purposes (see below), or we may need to refine our own thinking before we share it. (When considering a document’s purpose, the list above is handy, but certainly not exhaustive.)
What do you want readers to take away from this communication? The TL;DR formulation is popular for a reason. The exact framing may vary depending on the purpose of the communication, but an effective document should have a clear summary of some sort near the top.
What can we leave out, or link to elsewhere? The goal of communicating is not to share every fact we know, and all context is not created equal. Focus on the content that is necessary for achieving the communication’s purpose. Use comments or other documents to provide additional, optional context.

Depending on your comfort with the topic you’re writing about, and your comfort with writing itself, it could be hard to tailor your writing based on the answers to these questions, and that’s OK. Sometimes it makes sense to do some stream-of-consciousness writing before you move on to crafting your communication. Put another way: don’t let yourself get blocked by the need to answer these questions up front, but do press yourself to answer them, and refine your work as a result, before you press send.

Writing for the record #

Writing “for the record” is a distinct purpose with different considerations. A document created to record a moment in time — meeting notes, your personal reflections on a topic, or notes from a brainstorming session — can serve as an artifact of the moment. These documents don’t need to be particularly intentional or audience-driven, but that also means they are likely not a useful tool, on their own, for effective communication with others.

Another thing to keep in mind about moment-in-time documents is that their content can rapidly become outdated as teams, projects, and perspectives change. If you’re sharing these types of documents, make sure to prominently include the date when they were created or last known to be accurate.

Documents that intend to brainstorm about a topic are hugely valuable to the group directly involved in the brainstorming, but are very likely require synthesis before sharing with folks who weren’t directly involved. Share these in support of a communication that you write with purpose and intention, but probably not on their own.

Templates, patterns, and prior art #

Effective communication doesn’t have to be (and perhaps shouldn’t be) a creative writing exercise: it’s OK for your approach to be formulaic, for your writing to be dry and direct, and to draw inspiration from (or shamelessly copy) the things you’ve seen that work well.

Templates with predefined section headings and prompts can serve as a starting point for common types of communication, such as project briefs and design documents. For other types of communication, you can learn and lean on simple patterns like the tried-and-true, reader-centric inverted pyramid.

When you’re working on a piece of writing, take the time to review and reflect on examples of the same type of communication written by others. Who were they writing for? How did they organize their thoughts? How much did they write? What questions were you left with when you were done reading? What could they have left out?

A high-leverage activity that gets easier with practice #

Effective written communication is incredibly high-leverage: I’ve benefited countless times from reading a well-crafted document whose author couldn’t have known that I, in particular, would read it someday. I’ve conveyed complex topics to others with the push of a button, ensuring a follow-up synchronous conversation would be focused and informed. A document written with purpose and intention has a surprisingly long useful life, and that same document can scale in a way that in-person conversation cannot. When well-crafted documents are searchable and findable, their value increases even further.

None of this happens without effort, and that effort can feel like a lot if writing is outside of your comfort zone. If you aspire to be a leader who influences others and plays a pivotal role in big decisions, the effort is worth it — and it gets easier with practice. Embrace written communication opportunities when they present themselves, even if it’s a challenge, and create opportunities of your own. A regular practice of writing about the who and what and why of your work will, over time, pay big dividends in your ability to rapidly and effectively communicate with others.

Raspberry Pi setup notes

2020-12-28T00:00:00Z

The holidays are the perfect time for setting up a Raspberry Pi, screwing it up, and having to start over again. I’ve done this enough times in the last two weeks that I decided to write some stuff down.

Of note:

I’m using a Mac.
I've run these steps on a Raspberry Pi 4 and they worked.
I used the Pi Imager to put an image on an SD card. (If you don’t intend to use the desktop GUI, use the Lite image under “other.”)

Set up for headless use #

Before you put the SD card in the Pi, set it up so it will connect to wifi and you can connect to it via ssh. With the SD card still connected to the Mac:

cd /Volumes/boot
touch ssh

Set up wifi by creating the file wpa_supplicant.conf in /Volumes/boot, with the following contents:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=<2 letter ISO 3166-1 country code>

network={
 ssid="<network>"
 psk="<password>"
}

The country code for the US is US. (See other country codes.)

Connect & update & install an editor #

Figure out the Pi’s IP address.

nmap -sn 192.168.1.0/24

Then, ssh to it:

ssh pi@192.168.1.180

Enter raspberry as the password, and you’ll be logged in.

Change the password for the pi user:

passwd

You’ll hardly ever use this password, and maybe you’ll never use it, but if you do need to use it, you probably won’t remember it, and you’ll end up reading this blog post when you re-image your Pi.

Update package lists, upgrade packages (this could take a while), and install vim (if you don’t want to use nano).

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install vim

Create a new user #

Create a new user with sudo permissions:

sudo adduser rmurphey
sudo adduser rmurphey sudo

You’ll be prompted for a password. You’ll need to use this password every time that you sudo from the new account, which should make it more likely that you’ll remember it, but if your Pi use is seasonal like mine, good luck with that. I’m not saying you should write it down, but … you do you.

You could use this password to log in to your Pi, but that would be a pain. Instead, copy an ssh key from your Mac to the Pi so you can log in without a password:

ssh-copy-id rmurphey@192.168.1.180

Oh My ZSH #

Oh My ZSH is nice. Set it up.

sudo apt-get install zsh
sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

I like the fino theme.

Python 3 #

If you want to interact with devices connected to your Pi, you’ll probably end up using Python. Set up Python 3 as the default Python. Why isn’t it already? Who knows.

sudo pip3 install --upgrade setuptools
sudo apt-get install -y python3 git python3-pip
sudo update-alternatives --install /usr/bin/python python $(which python2) 1
sudo update-alternatives --install /usr/bin/python python $(which python3) 2
sudo update-alternatives --config python

Use sudo raspi-config to enable I2C and SPI.

Install some libraries for interacting with devices:

pip3 install RPI.GPIO
pip3 install adafruit-blinka

It seems like only the pi user has access to the SPI, I2C, and GPIO interfaces by default, so:

sudo adduser rmurphey spi
sudo adduser rmurphey gpio
sudo adduser rmurphey i2c

You might need to log out and log back in for these permissions changes to take effect.

CircuitPython on Raspberry Pi

Node #

JavaScript is fun, and johnny-five is a fun way to use JavaScript, so you’ll want Node.

Install with nvm:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash

Read the output; you’ll need to add something like this to .zshrc:

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion

Finally, install the version you want:

nvm install 14

Maybe #

sudo raspi-config to change the hostname.
sudo apt-get install screen so you can reconnect to a shell session.
sudo apt-get install motion if you want to set up a camera stream.

Couldn’t I just save a copy of the SD card before I screw it up again? #

Probably.

On micro-frontends

2020-11-19T00:00:00Z

My time at Indeed focused on improving front-end development practices and tooling across product teams. It was a journey from stabilization to remediation to modernization to, finally, innovation. In the process, I got to build and grow a team, and gain new skills in managing managers and setting product strategy and vision.

My capstone project, as it were, was a micro-frontend platform. Over the course of just a few quarters, it radically improved the speed with which teams could deliver and iterate on web UI, and radically shifted the ownership model of that UI. I wanted to spend a few minutes reflecting on that effort and some of the lessons I learned.

What the platform does #

The platform enables a “host” page to incorporate user interface and functionality that is provided by services called “providers.”

A host that implements the platform can specify regions where providers can be displayed within a page; at runtime, the platform 1) determines which providers might want to display in a given region, 2) brokers the request to those providers, and 3) converts their response into content that can be incorporated into the HTML response of the host page.

There are a lot of details hidden in those couple of sentences, but fundamentally, the system is incredibly simple: providers are just services that respond to HTTP requests with JSON that includes the initial HTML for the provider, along with URLs for the JavaScript and CSS for the provider. Hosts integrate a Java library that handles load balancing the requests to provider instances, and incorporate the HTML, CSS, and JavaScript from provider responses into the HTML returned by the server.

Technical challenges #

As we were building the system, some of the bigger technical challenges were around ensuring that product quality and functionality didn't suffer as we facilitated a new UI ownership model. We also needed to make sure that we could change the overall state of the platform without having to touch individual hosts or providers.

One of the most obvious technical challenges hearkened from my third-party JavaScript days. The browser’s runtime environment is shared among all of the code that runs within the page. That means that provider JavaScript or CSS can plausibly interfere with other JavaScript or CSS on the page — in an extreme case, provider JavaScript could delete the entire contents of a page. We had to balance legitimate provider needs with the imperative that a provider could not break critical functionality, which sometimes resulted in frustration for provider teams.

Resiliency was another big concern. If the the provider that displays job search results is slow or broken, job search results still need to display. The platform introduced the notion of “fallback content,” allowing providers to regularly register content that will be used if their provider fails to respond. This fallback content can include JavaScript that can fetch data and then render HTML in the browser, ensuring that users can still receive a functional UI even in the event of provider failure.

To mitigate the need for frequent host changes, we introduced a configuration artifact that could be updated without requiring a host deploy, and hosts would pick up the changes within minutes of the change being published. This artifact specified which providers should appear on a page, and where. With this mechanism, host pages could pre-define “zones” where providers were allowed to appear, and providers could be added and removed from those zones via the configuration. We later added the ability for hosts to revise the configuration at runtime, based on their knowledge of the context for the specific page view.

Non-technical challenges #

Just as we were rolling out the first prototype of the system, I had a realization: the biggest challenges we were going to encounter in securing broad adoption of a micro-frontend platform would be non-technical. Over the following two years, my hunch proved to be right.

Probably the biggest non-technical challenge was governance. When a single product team owned the entirety of a page, every decision about changes to that page went through that team. A micro-frontend platform fractured that ownership model; we had to develop processes that let the old owners feel confident that their pages wouldn’t change radically without their knowledge, while preserving the velocity gains for teams that now owned individual pieces of UI.

We talked a lot about “the pink button problem” — what would happen if a provider team violated a host team’s trust while trying to optimize for their own metrics? There were some technical guardrails we could impose, but it was easy to imagine behavior that would defy easy technical prevention. In reality, this largely proved to be a theoretical concern, not a practical one. Yes, there were some times that providers did surprising things, but because everyone worked at the same company and ultimately shared the same goals and mission, those situations were relatively easy to solve through conversation. Investing in technical solutions might prove necessary someday, but I’m glad we didn’t do it too early.

The other challenge that stands out was coaching provider teams on operations and ownership. From the beginning, we made sure that the platform ecosystem was highly observable: every provider and host team had real-time visibility into errors, latency, requests, cache misses, and more. What we quickly found is that provider teams were unaccustomed to worrying about these things, because the host teams had historically borne this responsibility.

While the platform is quite resilient to provider failure, the platform team continued to spend time escalating failures that provider teams weren’t noticing. It’s hard to know what the platform team can safely ignore — is that spike in provider errors due to a bug in the platform, or the provider? We never wanted a provider team to be the first to discover a bug that was ours, but we also couldn’t afford to be the first line of oeprational defense for provider teams. It’s a needle we still hadn’t figured out how to thread by the time I left.

To micro-frontend or not? #

The more successful the micro-frontend platform was, the more teams were coming to us and asking if they should use it — especially as they were creating whole new products and experiences. The answer was often “no,” because the Miroservice Premium applies to micro-frontends, too:

[M]icroservices introduce complexity on their own account. This adds a premium to a project’s cost and risk – one that often gets projects into serious trouble. – Martin Fowler

In my experience, a micro-frontend architecture has clear benefits when certain things are true.

First and foremost, a micro-frontend architecture seems to be most suited to an environment where web pages are deployed via a number of distinct codebases — that is, I’m not sure that a micro-frontend architecture makes sense in a monorepo. Distinct codebases alone aren’t a reason to move to a micro-frontend architecture. Here are some other indications that it might be appropriate:

There is agreement that independent product ownership of pieces of UI is acceptable for reasons other than delivery velocity. A micro-frontend approach makes sense if a product manager can draw a box around a thing and say, “I’m entirely OK if someone else makes decisions about what’s inside this box without talking to me, as long as this box continues to generally be focused on <concept>.”
There is substantial UI tech debt in the “host” codebase, and in-place remediation is broadly understood to be prohibitively expensive or disruptive. For example, in many legacy UI codebases, introducing the test automation required to enable continuous integration and delivery would require a wholesale rewrite of the UI. This argument is most compelling if there’s evidence that people have tried and failed to make progress in the past.
There is a strong case for the reuse of a piece of UI functionality across multiple surfaces. This UI functionality should be non-trivial and substantially similar across those surfaces. It’s possible for reuse to be managed via a library, but outside of a monorepo, this presents challenges of deployment coordination. Managing reuse via a service allows for centralized deployment of changes.

Unless at least one of these things is true, a micro-frontend approach may introduce complexity without benefit. In a greenfield project, perhaps the only reason to incorporate a micro-frontend architecture is for reuse — there is no tech debt yet, and there are other ways to achieve independent UI ownership without incurring the microservice complexity.

I strongly recommend againt incorporating a micro-frontend architecture until you can concretely describe the benefits you expect to achieve, and why those benefits are difficult or impossible to obtain otherwise. It can be a hard decision to undo once you head down the path.

Surprises and opportunities #

I always expected that provider teams would see velocity benefits, but I was surprised at the impact on host teams. Today, they’re largely spared from reviewing, releasing, and supporting external contributions, which has given them the bandwidth to invest in velocity improvements of their own. Host team velocity has more than doubled. Some host teams have chosen to author new functionality via providers rather than in the existing codebase, increasing their velocity even further.

One of the biggest unaddressed pain points is the reliance on libraries as a distribution mechanism for the platform itself. Provider and host teams tend to be risk-averse when it comes to updating libraries; “pinning” to a specific version is broadly viewed as acceptable, and there is no Indeed-standard mechanism for triggering upgrades. Driving adoption of a new version requires communication and, sometimes, cajoling. There’s an opportunity to deliver more value faster by shifting the bulk of the client library responsibility to a centralized service that the platform team can iterate on without requiring host deploys. On the other hand, this would introduce a new layer of resiliency risk that might not be acceptable.

While the platform we built was explicitly intended to be incorporated into existing applications, where those applications are ultimately responsible for serving the HTML for a page, I can definitely see a future where the platform is extended so that it is serving the pages, and those pages are composed of a lightweight shell — responsible for overall page layout and the page’s header and footer, say — and the rest of the page is then populated with providers.

In summary #

Two years of working on a micro-frontend platform gave me a ton of lived experience with the tradeoffs of a micro-frontend architecture. There's far more context than I can share here about why and how this approach was so well suited to deliver huge velocity improvements for UI development at Indeed — even small changes in that set of circumstances might have led to a very different result. Regardless, I hope that my experience might inform your own views on whether and how this approach might make sense for you.

Effective storytelling for internal platform teams

2020-11-18T00:00:00Z

Camille Fournier wrote a post a few months back about the challenges of product ownership for internal platforms. Her observations resonated with me a lot after four years working within an internal platforms organization at Indeed — how hard it can be to establish viable success metrics, how easy it can be to overestimate your understanding of your customers, how challenging it can be to serve a plausibly captive audience.

The post wrapped up with this (emphasis mine):

Great platform teams can tell a story about what they have built, what they are building, and why these products make the overall engineering team more effective. […] Without a clear strategy for showing impact and value, you end up overlooked and understaffed, and no amount of cool new technology will solve that problem.

Effective storytelling turns out to be one of the most important competencies for a platform team to develop. But how do you put it into practice? The exact implementation is going to vary a ton based on the communication norms of your company, but at a high level, here’s what’s worked for me and my teams.

Know your audience #

Effective storytelling starts with knowing your audience, and for an internal platform team, that audience can be broad. You might think about breaking it down like this:

Current users. These are the people who use your product today. They need to know about upcoming changes, and they’re your best partners in identifying new opportunities for your product and sharing your stories with others.
Future users. These are the people who have or will soon have the need your team is charged with meeting, but who haven’t started using your product. You want to make sure they know what you’re working on and how others are benefiting from your current offering.
The leadership that current, future, and former users answer to. Even the most fanatical user base will only get you so far — if you want to avoid the “overlooked and understaffed” fate, it helps to have your users’ leadership singing your praises too. To make that happen, they need to understand how you’re making life better for their teams.
Former users. These are the people who have used your product in the past, but they aren’t using it today. You probably don’t need to communicate with them en masse, but it’s good to be able to reach out to them as needed.
Your organizational leadership. Don’t forget to manage up — your own leadership needs to understand the value you’re creating and the problems you’re solving.

Ascertaining and segmenting the members of your audience — and then keeping those membership lists up to date — can be a challenge, especially in a large and/or rapidly growing organization. As an internal platform team, you should find yourself highly motivated to support efforts to create and maintain machine-readable org charts. You should also invest up front in instrumentation that lets you analyze who is using your tools and how often.

Barring a detailed understanding of who you need to communicate with, there’s always the all-hands mailing list or Slack channel, but at a certain size, communicating this way is about as effective as shouting into the void — even if almost everyone on the list is actually a user. You might be able to achieve results if your message is urgent and broadly applicable, but it’s no way to convey a nuanced, non-urgent update.

Communicate value, not effort #

About once a month for the last year, I sent out an newsletter to stakeholders and users of the platform my team owned. Every month, as I was preparing the newsletter, I would review the tickets that the team had worked on since the last newsletter went out. Every month, I would find myself a little bit surprised that the newsletter’s contents rarely ended up focusing on the work the team had completed.

Platforms, by their nature, tend to have a long tail of value delivery, and the value delivered can be annoyingly difficult to trace back to any single piece of work the team completed. A platform we started building in 2018 radically transformed the iteration speed for a business-critical team in 2020 — allowing that team to achieve some big experimentation wins faster as a result — but we did hardly any work in 2020 that was specifically in support of that team.

So while it’s important that your platform’s users know about new features, communicating about new features isn’t sufficient. In the worst case, it can stir up “what have you done for me lately?” feelings among users who don’t immediately benefit from the new features. Your storytelling strategy needs to put at least as much emphasis on the wins your users are achieving with the help of your platform. Those wins could be operational, like reductions in build times or outages; or business wins, like a set of winning experiments that a team was able to run and analyze faster thanks to your product.

When you’re communicating value, tell stories at the macro and the micro scale, and don’t be too reluctant to focus on outliers. “The business” cares that you reduced build times by 10%, or an average of 1 minute, but the AcmeWidget Team cares that, due to a peculiarity in their setup, your efforts actually cut their build times in half, by 15 minutes. Tell both stories. Get a quote from a dev on the AcmeWidget Team and feature it in your next newsletter.

Get creative in how you communicate #

Slack and email are the obvious candidates for communicating your team’s value story, but they’re just the start. The right venues for communication will depend heavily on your company culture, but here are some things I’ve tried that might give you new ideas:

Newsletters. Don’t just write an email: put in the effort to format your newsletter so it has a clear title, headings, and even images. Make sure it’s readable on mobile devices. Use a consistent subject line across “issues” of your newsletter. Anticipate forwarding: include a footer that tells readers where to subscribe to future issues. Use a mailing list so people can see past issues.
Slack. Have a public Slack channel for customer support. Forward your newsletters there using Slack’s email feature. Use consistent emojis to call attention to important announcements.
In-office advertising. Pre-pandemic, Indeed had large displays throughout the physical workspace, and we ran branded, multi-week campaigns to share platform success stories and invite future users to learn more about our platform.
Internal video podcast. I only did this once, and the primary audience was the team itself, but it was a unique storytelling tool. I can see using this venue a couple-few times a year to interview users, demo new features and use cases, and share your team’s future plans.
Internal presentations. This could be anything from a big tech talk to an ad hoc lunch and learn with a cobbled-together invite list. My experience is that there is enough hunger for understanding what platform teams are working on that I never needed much “permission” to set something up as long as attendance was optional.
Performance calibrations. This is a weird one, but I often found myself sharing my team’s value story during calibration conversations with other managers during the performance review cycle. Obviously this isn’t a primary purpose of calibrations, but it’s a good reminder to be prepared for unexpected opportunities.
Stakeholder conversations. The same artifact that you create for an internal presentation can serve you well for structured conversations with user teams and stakeholders, but it also never hurts to just drop them an email congratulating them on a win you heard about — and to gently connect the dots back to your platform.

Know your lines #

Telling your team’s value story requires having a consistent way of describing 1) why your team exists (your mission), and 2) what’s plausibly at the end of the long arc your team is traveling (your vision). Will Larson does a good job of defining the component parts of a vision and the steps you can take to create a vision document, but you’ll also want a short, pithy version that you can include in communications to your users. At Indeed, my team’s vision boiled down to this:

Product teams building user interface can focus on the unique business value they are trying to deliver; everything else just works.

Our mission was equally simple:

We provide crucial capabilities that allow product teams to build and iterate on user interface — successfully, autonomously, and without regret.

We weaved these words, or echoes of them, through every communication about the work we were doing and why it mattered. They featured in every presentation to the team, to leadership, and to the broader organization. While it’s unlikely that anyone outside of your team will be able to recite your mission and vision verbatim, the constant repetition of the words and themes should serve to establish an ambient understanding of the value you’re providing.

A relentless repetition of your mission and vision also helps your team stay connected with the why of the work they’re doing, which brings us to the next part …

Everyone has a part to play #

Even if you’re so lucky as to have a person whose full-time job is internal marketing — and let’s be honest, what we’re talking about here is certainly a flavor of marketing, and certainly time-consuming — everyone on the team should be able to explain why the work the team is doing is valuable. Engineers, QA, product managers, project managers — everyone should be on the lookout for stories to tell about the value the team is creating. Everyone should be attuned to opportunities for the platform to create new value, in line with the team’s mission and vision.

Succeeding on a platform team will frequently require direct engagement with a larger portion of users than one might encounter on a traditional product team — especially at a company big enough to require a platform team in the first place. The best of these are centered in understanding the value that user is trying to achieve, and in turn help the user understand the part the platform is playing in delivering that value.

Direct user interactions — via support channels, office hours, fireside chats, user research, or any other situation where you’re talking directly to a user or a team — are golden opportunities. Well executed, they cultivate evangelists for your product, people who will help the broader organization understand your product’s value with little effort on your part. Poorly executed, these interactions can do damage to even the best value story.

Make sure your team members see the straight line that connects their users’ success with their own. Highlight user success in team channels, and encourage individual team members to follow up on the usage of the things they build.

Wow, that’s a lot. #

Yes, and I also feel like this barely scratches the surface of the work of telling your team’s story. Unless you do find your team staffed with a person who’s directly responsible for this type of communication, it really does need to be a team effort, and your team’s value story needs to be a regular part of planning and retrospective conversations.

It could be tempting to point at your team’s product manager (if you have one) as the person who should carry most of this weight. Their role here isn’t small, but it’s also not sufficient — especially on a team where other engineers are the product’s customers. Engineers and engineering managers all need to be looking for ways to engage in telling their team’s value story. That doesn’t mean they each need to be a public face of the team, but it does mean they need to be thinking about the why, the measurement, and the communication needs for the initiatives the team is working on. (Relatedly: A platform team might not be a comfortable home for an engineer who just wants to be heads-down in the code.)

If all of this feels overwhelming and you’re looking for somewhere to start, it’s here: make your team’s value story a part of your team’s everyday conversations. Incorporate customer engagement into your planning and development cadence. Encourage your team to ask challenging questions — about the expected value of the work they’re doing, about which customers are expected to benefit, about how customers are thinking about the product. As your team starts to understand how its success is connected with the answers to those questions, the rest will start to follow.

Notes on setting up a JS project, circa 2016: ESLint

2016-01-20T00:00:00Z

This post is really old! I've kept it around because it may still be interesting, but many things may be out of date.

I wrote previously about setting up a client-side JS project to use Webpack and Babel. In this post, I'll cover the next step: adding ESLint to the project to keep the JavaScript code clean and following a consistent style.

This is the easiest step of all — so easy that at first I felt silly writing a post about it, but there were a couple of things worth noting.

To start, I installed ESLint to the project using npm install --save-dev eslint. Next, from the root directory of the project, I asked ESLint to create an initial configuration file for me:

./node_modules/eslint/bin/eslint.js --init

This gave me the option of using one of several popular ESLint configurations, or generating my own. I decided to generate my own, which took me through the following prompts:

? How would you like to configure ESLint? Answer questions about your style
? What style of indentation do you use? Spaces
? What quotes do you use for strings? Single
? What line endings do you use? Unix
? Do you require semicolons? Yes
? Are you using ECMAScript 6 features? Yes
? Where will your code run? Node, Browser
? Do you use JSX? No
? What format do you want your config file to be in? JavaScript
Successfully created .eslintrc.js file in /Users/rmurphey/personal/js-games

When I was done, like it says above, I had an .eslintrc.js file in the root of my project. It looked like this:

module.exports = {
    "rules": {
        "indent": [
            2,
            4
        ],
        "quotes": [
            2,
            "single"
        ],
        "linebreak-style": [
            2,
            "unix"
        ],
        "semi": [
            2,
            "always"
        ]
    },
    "env": {
        "es6": true,
        "node": true,
        "browser": true
    },
    "extends": "eslint:recommended"
};

The generated file was indented at four spaces, and that's when I realized that I hadn't been asked about my preferred indentation width, just whether I wanted tabs or spaces. I adjusted the indent rule to enforce two-space indentation ... and re-indented the file:

// ...
"indent": [
  2,
  2
],
// ...

In reading the documentation, I learned that even though my config file said I wanted ES6 support, this support didn't include ES6 modules — if I used import or export in my code, ESLint would treat it as a syntax error. To allow modules, I needed to add another property to my config: ecmaFeatures.

// ...
"ecmaFeatures": {
  "modules": true
}
// ...

It's also worth noting one of the last lines in the file: "extends": "eslint:recommended". ESLint has a set of recommended rules — such as disallowing console statements — that are automatically enforced when this extends is present. I can override them if I want by setting new values for those rules in my .eslintrc.js file.

I wanted to try out my setup, but I didn't want to keep typing the full path to the ESLint executable, so I added a new npm script to lint all the files in the client/ directory:

"lint": "eslint client/**"

When I ran npm run lint, though, I got a slew of errors about files in client/dist/. This directory is where my Webpack build goes, and I don't actually care about linting that code, because it's uglified. To deal with this, I created an .eslintignore file and added client/dist/* to. With that change, running npm run lint just showed me a few errors for the files I care about in the client/ directory.

There were a couple more changes I wanted to make to my ESLInt setup. For one, ESLint outputs a warning about the fact that I've ignored some files. I decided to live dangerously and suppress the warnings by passing --quiet to the eslint command. Secondly, I know from experience that ESLint can get slow on large code bases; the --cache option addresses that. The --cache option causes ESLint to create an .eslintcache file, so I added that to my .gitignore file as well.

None of this would be very valuable if I had to remember to lint my files whenever I make a change. There are editor plugins that will read my .eslintrc.js file and show me errors, but I wanted to make sure I couldn't commit code that doesn't pass ESLint. The ghooks module lets me make sure ESLint runs as a pre-commit hook; I installed it with npm install --save-dev ghooks, and then added a few lines to my package.json:

"config": {
  "ghooks": {
    "pre-commit": "npm run lint"
  }
}

Closing Thoughts #

I love how well the ESLint rules are documented, and the error output of ESLint makes it easy to find the documentation when you want to learn more about the rule that your code has violated. ESLint also makes it easy to override a rule for a file or even for a single line. Adding ghooks to the mix allows for future improvements, such as enforcing commit message formats, running tests pre-push, etc.

You can see the code as of this post here.

Notes on setting up a JS project, circa 2016: Babel and ES6

2016-01-20T00:00:00Z

This post is really old! I've kept it around because it may still be interesting, but many things may be out of date.

I wrote previously about setting up a client-side JS project to use Webpack. You can see the code as it stands as of that last post. In this post, I'll cover the next step: using Webpack and Babel to convert ES6 code into code that can be used in all modern browsers.

Why ES6? I'm pretty late to the ES6 bandwagon, but after using it on a couple of projects recently, I'm sold just based on the convenience of fat arrow functions, block scoping, constants and destructuring — nevermind the features that go far beyond sugar.

Thankfully, using ES6 in a project that already uses Webpack is straightforward, though it does require installing a lot of things.

As of the last post, my webpack.config.js looked like this:

var path = require('path');

module.exports = {
  context : __dirname,
  entry : {
    'number-guessing' : './number-guessing/index'
  },
  output : {
    path : path.join(__dirname, 'dist'),
    filename : "[name].js"
  },
  devServer : {
    contentBase : __dirname
  }
};

I also had a JS file (client/number-guessing/index.js) that had a simple console.log statement in it. I needed to modify this JS file to have some ES6 goodness in it, in order to test whether our new Webpack setup works:

let foo = () => {
  console.log('it works');
};

foo();

Before making any changes to my Webpack config, I checked to see what the output would be if I ran my current build script, npm run build. Surprisingly, this worked, even though I hadn't done any Webpack setup to deal with ES6 code. When I looked at the output in client/dist/number-guessing.js, I saw at the end of the file that the code in client/number-guessing/index.js hadn't been transpiled down to ES5 — the fat-arrow function was still there. I knew that, when my Babel config was working, I'd see the fat-arrow function converted into a normal ES5 function.

Configuring Webpack to use Babel to transform my ES6 code was pretty straightforward. First, I needed to install the Babel basics:

npm install --save-dev babel-core babel-loader babel-preset-es2015

Next, I added a module.loaders property to my Webpack config. This new section tells Webpack to use the Babel loader on any JS file, except for JS files in the node_modules directory. Importantly, the Babel loader doesn't do anything by default — you have to tell it to use the es2015 preset to convert code from ES6 (ES2015) to ES5.

var path = require('path');

module.exports = {
  context : __dirname,
  entry : {
    'number-guessing' : './number-guessing/index'
  },
  output : {
    path : path.join(__dirname, 'dist'),
    filename : "[name].js"
  },
  module : {
    loaders : [
      {
        test : /.js$/,
        exclude : /node_modules/,
        loader : 'babel',
        query : {
          presets : [
            'es2015'
          ]
        }
      }
    ]
  },
  devServer : {
    contentBase : __dirname
  }
};

Now, when I run npm run build, I can see at the end of the output in dist/number-guessing.js that the fat-arrow function has been converted to an ES5 function; my Babel loader is working. However, it is really slow — far slower than my old, Babel-free build. To make this a bit better for local development, I set the cacheDirectory setting for Babel to true as long as the code isn't running in a production environment:

var path = require('path');

module.exports = {
  context : __dirname,
  entry : {
    'number-guessing' : './number-guessing/index'
  },
  output : {
    path : path.join(__dirname, 'dist'),
    filename : "[name].js"
  },
  module : {
    loaders : [
      {
        test : /.js$/,
        exclude : /node_modules/,
        loader : 'babel',
        query : {
          cacheDirectory : !(process.env.NODE_ENV === 'production'),
          presets : [
            'es2015'
          ]
        }
      }
    ]
  },
  devServer : {
    contentBase : __dirname
  }
};

Now that I have Webpack set up to transpile ES6 code to ES5, I can use all of the ES6 goodness — classes, imports, exports, and much more.

Closing Thoughts #

It's annoying that the Babel loader doesn't do anything out of the box — this is a change from Babel 5 to Babel 6. Other than that, getting Webpack to transform ES6 code to ES5 is pretty straightforward — at least until you try to write tests, which is a topic for another post.

You can see the code as of this post here.

Notes on setting up a JS project, circa 2016: Webpack

2016-01-19T00:00:00Z

I'm working on a project that may or may not see the light of day: a collection of simple games written in JavaScript. The goal of the project is to help JavaScript learners understand how to break moderately complex problems into their constituent parts; in the process, the project will also show the use of modern JavaScript tooling. A secondary goal for me is to write about the process of working on the project; this post is an attempt at that.

The project is going to consist of both code and content, so, to start, I created a content/ directory and a client/ directory. I know that one of the games I want to show will be a simple number-guessing game, so I made a number-guessing/ directory. I also ran npm init to generate a package.json file. When I was done with this initial setup, this is what my files looked like:

/js-games
  /client
    /number-guessing
  /content
  package.json

I knew that I wanted to use Webpack, and I wanted to make sure I got it set up before I got too far with anything else. To verify that Webpack was working, I'd need some basic JS and HTML. I created a file client/number-guessing/index.js, and put a simple console.log('it works') inside; I also created a file client/number-guessing/index.html, and added the following:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Number Guessing</title>
  </head>
  <body>
    <script src="./number-guessing.js"></script>
  </body>
</html>

With these pieces in place, I was ready to take a stab at configuring Webpack.

Setting up Webpack #

First, I needed to install the Webpack npm module, along with the Webpack development server:

npm install --save-dev webpack webpack-dev-server

Next, I needed to create my Webpack config. Traditionally, this file goes in the root directory of a project, alongside the package.json; however, for a project like this, I felt like it made more sense for the Webpack configuration file to be with the client files, so I created client/webpack.config.js.

/js-games
  /client
    /number-guessing
      index.html
      index.js
    webpack.config.js
  /content
  package.json

I knew that I would want to create a separate bundle for each game, so I would need multiple "entry points" in my Webpack config, one for each game's index.js file. With that in mind, this was my first Webpack config:

var path = require('path');

module.exports = {
  entry : {
    'number-guessing' : './number-guessing/index'
  },
  output : {
    path : path.join(__dirname, 'dist'),
    filename : "[name].js"
  }
};

Before I went any farther, I wanted to try it out. First, I ran cd client to move into the client/ directory. Then, from the client directory, I ran:

../node_modules/webpack/bin/webpack.js

This was the output:

-> % ../node_modules/webpack/bin/webpack.js
Hash: 3dec0922a524080dea35
Version: webpack 1.12.11
Time: 41ms
             Asset     Size  Chunks             Chunk Names
number-guessing.js  1.41 kB       0  [emitted]  number-guessing
   [0] ./number-guessing/index.js 25 bytes {0} [built]

Seems good. I opened the file client/dist/number-guessing.js and inspected it; after the Webpack loader code, at the very end of the file I saw my console.log statement. My simple file had been built as expected; later, I could use that file to load other modules, and configure Webpack to Uglify the output, transpile ES6 code to ES5, and more.

Setting up the Webpack dev server #

Next, I wanted to set up the Webpack development server. This server serves static files from your filesystem, and also watches your JS files, updating the build whenever there are changes. I wanted to use it to serve the HTML file, which would in turn load the built version of my JS.

Still in the client directory, I ran:

../node_modules/webpack-dev-server/bin/webpack-dev-server.js

This was the output:

-> % ../node_modules/webpack-dev-server/bin/webpack-dev-server.js
http://localhost:8080/webpack-dev-server/
webpack result is served from /
content is served from /Users/rmurphey/personal/js-games/client
Hash: 3dec0922a524080dea35
Version: webpack 1.12.11
Time: 71ms
             Asset     Size  Chunks             Chunk Names
number-guessing.js  1.41 kB       0  [emitted]  number-guessing
chunk    {0} number-guessing.js (number-guessing) 25 bytes [rendered]
    [0] ./number-guessing/index.js 25 bytes {0} [built]
webpack: bundle is now VALID.

I opened the URL from the output (http://localhost:8080/webpack-dev-server/), which presented me with a simple UI provided by the Webpack server. I clicked on the number-guessing link, and when the page loaded, I saw my message in the console. However, the URL was still http://localhost:8080/webpack-dev-server/, and my actual HTML wasn't being loaded.

I wanted to be able to access my test page directly, so I tried navigating to http://localhost:8080/number-guessing/. It loaded, but there was now an error in the console: the page was trying to access http://localhost:8080/number-guessing/number-guessing.js, and was getting a 404 in response. It seemed that if I wanted to be able to access my test page directly, I was going to need to change the reference to my script in the HTML.

Looking back at the output from when I started the Webpack server, I saw webpack result is served from / -- this means that my "built" JavaScript was served from the server root. I tried loading http://localhost:8080/number-guessing.js and, indeed, it worked. I changed the script tag in my HTML to reflect the actual location of my built JavaScript bundle, one level up from the HTML file:

<script src="../number-guessing.js"></script>

With this change, the number-guessing HTML now worked whether I accessed it via the http://localhost:8080/number-guessing/ URL or by clicking on the link in the Webpack server UI.

Adding an npm script #

So far, I had been running Webpack commands directly. This project may eventually need a tool like gulp or grunt for task automation, but for now I figured I would just use an npm script as a shortcut for the command to start the server.

I edited my project's package.json to add a new "serve" entry to the "scripts" object:

"serve": "webpack-dev-server --config ./client/webpack.config.js",

In an npm script, I can skip providing the full path to an executable; npm knows to look in the right places to find webpack-dev-server. Since my config was in a non-standard location, I had to pass its location to the command.

When I ran npm run serve, this was the output:

-> % npm run serve

> js-games@0.0.1 serve /Users/rmurphey/personal/js-games
> webpack-dev-server --config ./client/webpack.config.js

Hash: b74b27e56bc0a032a890
Version: webpack 1.12.11
Time: 28ms

ERROR in Entry module not found: Error: Cannot resolve 'file' or 'directory' ./number-guessing/index in /Users/rmurphey/personal/js-games
webpack: bundle is now VALID.
http://localhost:8080/webpack-dev-server/
webpack result is served from /
content is served from /Users/rmurphey/personal/js-games

Though Webpack said my bundle was "VALID", the error on the line before made clear that something was wrong: Webpack was looking for the client/number-guessing/index.js file in the wrong place: in number-guessing/index.js instead. Even though my webpack.config.js file was in the client/ directory, it seemed Webpack was looking for the file one level up, where package.json was. Even running npm run serve from inside the client/ directory didn't change this.

To address this, I needed to tell Webpack where to start its search, using the context configuration option. I modified my webpack.config.js to add a context:

var path = require('path');

module.exports = {
  context : __dirname,
  entry : {
    'number-guessing' : './number-guessing/index'
  },
  output : {
    path : path.join(__dirname, 'dist'),
    filename : "[name].js"
  }
};

With this change, the build was working, but now when I opened the server in the browser, I saw that my project's root directory was being served -- I only wanted the content/ directory to be served. Fixing this required another change to my Webpack config, to configure the dev server to use a different directory as its "content base":

var path = require('path');

module.exports = {
  context : __dirname,
  entry : {
    'number-guessing' : './number-guessing/index'
  },
  output : {
    path : path.join(__dirname, 'dist'),
    filename : "[name].js"
  },
  devServer : {
    contentBase : __dirname
  }
};

With this change, my npm script was now showing the same thing I was seeing earlier, when I ran the command to start the dev server directly.

I made three more small changes: adding the --hot option, to enable hot module replacement; adding the --open option, to automatically open a browser to the dev server whenever I run it; and adding the --inline option to automatically reload the page when I make changes to the JavaScript:

"serve": "webpack-dev-server --hot --open --inline --config ./client/webpack.config.js"

Lastly, I added a script to generate a build:

"build": "webpack --config ./client/webpack.config.js"

Closing Thoughts #

Setting this all up took less time than writing this post, but that's due largely to the fact that I've done this kind of setup several times before. I knew what I needed, and past experience gave me good instincts when something wasn't working quite like I hoped. The Webpack docs are pretty good if you already know what you're doing, but I can imagine they're painfully opaque if you don't.

If you don't want to use one of the many boilerplates that exist, then my main advice would be: "baby steps." Get a tiny thing working, and commit that; then move on to the next tiny thing. It's the process I followed in the setup I outlined above, and it helped me have confidence each step of the way.

You can see the code as of this post here.

Indeed

2016-01-04T00:00:00Z

It was three years ago today that I sat for a day of whiteboard interviews in a tiny room at Bazaarvoice with a series of terrifyingly smart people. When the interviews were over, we had cupcakes — it happened to be my birthday. A few days later, I had an offer; a few weeks later, I flew back to Austin for my first day, staying at a rental house where the chickens who greeted me were like a warm sendoff from my Bocoup friends. A year later, after working remotely and making the trip back and forth every few weeks, my family and I moved to Austin.

The last time I had the same job for three years, I was still working in print, and so it's a remarkable sort of thing to look back at all I've learned since that day three years ago. These have been among the most transformative years in my software career, full of profound lessons in what it means to create, support, and maintain software at scale, with all the rich experience and inevitable technical debt that comes with a company that's been in business since before I started working on the web.

I've worn several hats at Bazaarvoice — software engineer, team lead, inadvertent manager. I shaped the technical underpinnings of brand new products; stabilized and hardened products that were built to meet a deadline, with the assumption that quality and maintainability would be bolted on later; watched products that were assumed to have a shelf life of months live on for years; and helped improve the operational stability of an acquired product while melding it with the Bazaarvoice ecosystem. I made plenty of mistakes, too — technical mistakes, strategic mistakes, people mistakes. None of them are appropriate fodder for a public retrospective, of course — let's all just agree that manually deploying custom code for your biggest customer is as bad an idea as it sounds — but I hope I've learned from them enough that I'll manage not to repeat them.

Where I succeeded, it was never on my own. I was extremely lucky to be able to recruit a handful of talented JavaScript friends early on — first Brian Sinclair, then Dan Heberden and Lon Ingram — and together with great managers and peers, we grew a powerful front-end team. Those terrifyingly smart people who interviewed me were welcoming and helpful and kind from day one, granting me a seat at the big kids' table long before I had earned it, helping me survive and thrive even when I barely knew how to use a metaphorical fork.

I thought I knew a lot of things when I started: that whiteboard interviews were universally useless; that CS degrees were generally overrated; that client-side developers didn't need to worry about algorithmic complexity; that I could continue to excel as a front-end developer without concerning myself with where the data came from; that client-side applications were the future; that devops was someone else's job; that Java still existed mostly so we could make fun of AbstractFactoryClassBeans.

Somewhere along the way, I stopped feeling quite so precious and exceptional about being a front-end developer, and started to embrace being a software engineer who happened to understand the front end better than most. And one day I realized I was actually becoming a big kid: solving problems across teams and products; connecting dots that I was uniquely qualified to connect; developing and operationalizing systems and tools that extended far beyond the browser; and communicating clearly and persuasively about challenges and tradeoffs and solutions. Eventually, I earned the right to claim a seat at that big kids' table without apology, and I hope I have treated that seat with its due respect.

It's time for me to hit the reset button, to visit the world outside my new comfort zone, and see what the next three years have to teach me. I'll say goodbye to the amazing folks at Bazaarvoice in mid January, take a month off, visit a hero, hopefully do some writing. My new role, tackling the front end at Indeed, starts in February.

Speaker economics

2016-01-02T00:00:00Z

This post is really old! I've kept it around because it may still be interesting, but many things may be out of date.

Camille Fournier tweeted earlier today asking about the economics of conferences, and how that impacts whether speakers get paid to speak. This is something I've thought about a lot over the years, so I wanted to write a few things down.

I have the tiniest bit of rather outdated experience with running a couple of conferences; neither made a profit (intentionally), and both had low ticket prices and pretty decent lineups (especially considering the ticket prices). Speaker travel and accommodation costs were covered, though for one of the events, the speakers were friends who crashed at my house. In my limited experience, running a "community" conference is risky, exhausting, rewarding, and more expensive than you want it to be. Still, I feel strongly that it borders on unethical to not at least cover speaker costs. No one has a "right" to run a conference, and if you have to make speakers pay their own way, you shouldn't be running one. Find more sponsors, raise ticket prices, fire the hotel that charges you $5 for a can of soda — but figure it out.

Most of my conference experience is as a speaker. I've received small honoraria for speaking at a few events, but otherwise, my speaking has been unpaid beyond having airfare and accommodation covered. When traveling for conferences, I pay for many of my own meals, and almost always pay for airport transportation, parking, and other miscellaneous expenses. I am well paid, so none of this is a hardship, but that's not true for everyone. For most of my speaking career, it has been part of the terms of my employment that I could travel to speak at conferences without using vacation time. In return, I mention my employer at the beginning and end of each talk.

It may seem that whether speakers get paid is a function of a conference's budget, but I have a hunch that it's a matter of another sort of economics. I think conferences don't pay speakers because conferences don't have to pay speakers. Supply so exceeds demand — see, for example, the 279 proposals that CascadiaFest received for approximately 30 speaking slots — that there is little incentive for conferences to pay. Indeed, the incentive seems to be the opposite — there are so many people willing to speak that new events are popping up constantly.

"But conferences want good speakers," you say. It's devilishly difficult to define and identify "good"; speakers who have been stars at past events may be duds at future ones, and no-name speakers may bring down the house, cementing an event's credibility for years to come. Complicating things further, the speaker you think is a disappointment can be another attendee's favorite — people are weird that way. Conferences can create an overall experience that dulls the impact of bad speakers, and they can mentor inexperienced speakers to further reduce the risk.

Certainly, there are speakers whose presence in the lineup will speed ticket sales with the promise of both entertainment and quality content, ensuring critical cash flow to pay the venues and vendors who want money up front. Maybe those speakers should get paid? The problem with that: they often already are paid, by companies who have noticed their talent for speaking and communicating and have hired them to do exactly that. Their presence on the speaking circuit — and their companies' (often unspoken) sponsorship of that presence — serves to further ensure that there's ample supply of free speakers (often of the young-ish white male variety).

Speaking at conferences altered the trajectory of my career, but the professional benefits for me are now incremental at best, and there are other ways to make an impact on — and remain visible in — the community. I've made my peace with not getting paid by making sure my employer places some monetary value on my speaking — that is, that they let me save vacation time for vacation. I also choose conferences that take me to places I want to go, to see people I want to see. I long ago decided to feel no shame in developing talks with a long shelf life, rather than trying to come up with something new for every event. Finally, I've stopped feeling obligated to immerse myself in the conference experience — for $0, my obligation is fulfilled when I get off stage, and there's a good chance you'll find me out to dinner with a friend rather than at the after-party.

And that's the unfortunate part: attendees may think their ticket includes time with the speakers, but many popular but unpaid speakers eventually stop making that part of the deal. There's only so much of your time and energy you can give away for free, even (especially?) when there's an open bar.

I don't think it's fair to speakers that they spend so much time — time working on their talks, time away from family, time away from work — without compensation. I think this unfairness particularly impacts speakers who are minorities in the tech world and already, in aggregate, at a financial and political disadvantage. As Camille points out, though, the response to a request for compensation is generally dead air. There are simply too many other good — or at least good-enough — options.

Chrome HTTP/2 Log Parser

2015-11-28T00:00:00Z

This post is really old! I've kept it around because it may still be interesting, but many things may be out of date.

When I wrote about HTTP/2 earlier this week, one of my complaints was that there wasn't a great way to visualize what was going on with an HTTP/2 connection — and that Chrome's dev tools often tell a story that seems to conflict with the story told by Chrome's net-internals log.

I spoke with Andy Davies the day after I published that post, and he pointed me to a tweet from Moto Ishizawa from October, showing a screenshot of an "HTTP/2 Stream Analyzer." As far as I can tell, the thing in the screenshot hasn't been released. I hope it will be, but in the meantime I've written chrome-http2-log-parser, a Node module that takes the text from Chrome's net-internals HTTP/2 tab and spits out more useful data, including some basic text visualizations.

If you're not familiar, the output of Chrome's HTTP/2 net-internals log, here's a little bit of one:

t=    6952 [st=       0]    HTTP2_STREAM_UPDATE_RECV_WINDOW
                            --> delta = 15663105
                            --> window_size = 15728640
t=    6952 [st=       0]    HTTP2_SESSION_SENT_WINDOW_UPDATE_FRAME
                            --> delta = 15663105
                            --> stream_id = 0
t=    6953 [st=       1]    HTTP2_SESSION_SEND_HEADERS
                            --> fin = true
                            --> :authority: 52.23.178.48
                                :method: GET
                                :path: /index-separate.html?push=/common/libs/combined.js
                                :scheme: https
                                accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
                                accept-encoding: gzip, deflate, sdch
                                accept-language: en-US,en;q=0.8
                                upgrade-insecure-requests: 1
                                user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
                            --> priority = 4
                            --> stream_id = 1
                            --> unidirectional = false
t=    7020 [st=      68]    HTTP2_SESSION_RECV_SETTINGS
                            --> clear_persisted = false
                            --> host = "52.23.178.48:443"
t=    7020 [st=      68]    HTTP2_SESSION_RECV_SETTING
                            --> flags = 0
                            --> id = 2
                            --> value = 0

This goes on for hundreds of lines, making it essentially impossible to visualize what's happening.

The chrome-http2-log-parser module reads a file that contains this log and turns it into:

An array of objects, one for each log entry.
An object with a property for each stream ID, where the value of the property is an array of objects representing each log entry for the stream.

It also offers two text-based reporters (I'm hoping to also make an HTML reporter soon); they provide time-based visualizations of the data at a resolution of your choosing:

0:	             *
1:	S            R                     D         D
2:	             PDR       DDDD       DDDDDD   DDDDDD                                  A
3:	                                      S          DRDDDDDDDDDDDD
5:	                                        S             R  DDDDDDDDDD                              DDDD          D
7:	                                        S               R     DDDDDDDD                           DDD           D
9:	                                        S               R        DDDDDDDD                        DDD           D
11:	                                        S                R         DDDDDD  DD                    DDD           D
13:	                                        S                R                  DDDDDDDD              DDDD         D
15:	                                        S                R                     DDDDDDDD            DDD         D
17:	                                        S                R                       DDDDDDDD          DDD         D
19:	                                        S                R                         DDDDD       DDDDDDDD        D
21:	                                        S                R                                     DDDDDDDDDDDDDD  DD

You can grab it on npm — note that you'll need Node 4.2 or greater:

npm install chrome-http2-log-parser

If this is useful to you, I'd love to hear about it; likewise if you have ideas about how to make it better. I've written up a couple of TODO items at the end of the README, if you're so inclined :)

Building for HTTP/2

2015-11-25T00:00:00Z

This post is really old! I've kept it around because it may still be interesting, but many things may be out of date.

Earlier this year, I got the chance to speak with Google's Ilya Grigorik about HTTP/2 for the 1.10 episode of the TTL Podcast. It was a great primer for me on how HTTP/2 works and what it means for how we build the web, but it wasn't until more recently that I started to think about what it means for how we build the web — that is, how we generate and deploy the HTML, CSS, and JS that power web applications.

If you're not familiar with HTTP/2, the basics are simultaneously simple and mind-boggling. Whereas its predecessors allowed each connection to a server to serve only one request at a time, HTTP/2 allows a connection to serve multiple requests simultaneously. A connection can also be used for a server to push a resource to a client — a protocol-level replacement for the technique we currently call “inlining.”

This is everything-you-thought-you-knew-is-wrong kind of stuff. In an HTTP/2 world, there are few benefits to concatenating a bunch of JS files together, and in many cases the practice will be actively harmful. Domain sharding becomes an anti-pattern. Throwing a bunch of <script> tags in your HTML is suddenly not a laughably terrible idea. Inlining of resources is a thing of the past. Browser caching — and cache busting — can occur on a per-module basis.

What does this mean for how we build and deploy applications? Let's start by looking at the state of the art in client-side application deployment prior to HTTP/2.

Deploying JavaScript Applications (2013) #

In March of 2013, Alex Sexton wrote Deploying JavaScript Applications, and it's what I consider to be the canonical post on the topic for sites and apps that include more than about 50K of client-side code.

In his post, Alex describes a deployment that uses a "scout" approach: a small bit of code, included directly in the HTML or loaded via <script> tag.

The scout file exists to balance the desire for application resources to be highly cacheable vs. the need for changes to those resources to take effect quickly.

To meet that goal, the scout needs a short cache time when it's a file; if the scout is in the HTML, then the HTML itself needs a short cache time. The scout contains information about the location of the file(s) that provide the current version of the application, and the code necessary to load those files.

Files loaded by the scout can have extremely long cache times because the scout loads resources from versioned URLs: when a resource is updated, it is hosted at a new URL, and the scout is updated to load the resource from that new URL.

Why a scout approach rather than just loading the versioned files using <script> tags directly from the HTML? The scout technique lets you deploy changes to your JavaScript application without requiring a re-deploy of the server-side application. (In an ideal world this might not seem valuable, but in the real world, it often is.) When the scout is served separately from the HTML, it also allows for a different caching strategy for the HTML.

In this system, it's typical that the scout would load one or two JavaScript files that were generated by combining the modules needed for the initial state of the application. More code might be loaded later to support additional application behavior; again, that code would typically comprise a set of modules shipped in a single file.

There are a few shortcomings inherent to this approach, which are difficult to overcome without upsetting the balance between cacheability and changeability:

Shipping the application as a large file with a long cache time works great for repeat visitors, but not so well for first-timers who have to wait for the large file to load.
All users have to download the whole large file again whenever something changes — even something small.
Even when nothing changes, a short cache time means repeat visitors may end up re-downloading the scout frequently.

Adding HTTP/2 to the mix — that is, flipping the switch that gets your server to start speaking HTTP/2 to browsers that understand it — has a nominal positive impact on the performance of an app crafted for maximum performance on HTTP/1. Indeed, the applications most likely to see big improvements without big changes are applications whose deployments were poorly designed in the first place.

To see performance gains in a well-engineered deployment, we'll have to re-engineer the deployment itself.

Splitting it up #

One of the most obvious opportunities is presented by HTTP/2's ability to handle multiple requests over the same connection. Rather than shipping a single large application file over the wire, what if we tell the scout to load the individual modules that make up the application? We would no longer have to invalidate the cache for the whole application every time we make a change.

A few reasons come to mind why this might be a bad idea.

The first is the concern that compression might suffer if shipping modules individually. As it turns out, though, combining multiple modules into a single file results in only slightly better compression than if the modules are compressed individually. For example, compressing a file containing minified versions of jQuery, Underscore, and Backbone results in 42,186-byte file; compressing each minified file individually results in a combined size of 42,975 bytes. The difference is 789 bytes -- barely meaningful.

Other second concern may be more legitimate: our server or CDN may be unhappy about serving one request per module; and it may be unduly complex to ship a single module per file, especially since any given request might fail for whatever reason. For the sake of discussion, we'll assume that it's reasonable to do some grouping of modules into individual files.

How to group those modules is up for debate. One strategy could be to group files according to their likelihood of changing, recognizing that library and framework modules don't change often, while application modules do. Another strategy would be to group files associated with a unit of useful functionality, though this leaves us needing a way to deliver code that's shared across units of functionality.

At Bazaarvoice, we solve this concern via a lightweight require/define system that ships in the scout file, allowing us to share vendor files such as jQuery and Backbone across applications. An application can express a dependency on a vendor file using NAMESPACE.require(), and vendor files declare themselves using NAMESPACE.define(). Once a vendor file has been defined, other modules on the page have access to it immediately via NAMESPACE.require().

Versioning #

For HTTP/1.1-friendly builds, we always increment the version of the built application file, and embed a URL pointing to that new version in the scout file. We do this because it is essentially guaranteed that the contents of the application file have changed whenever we do a new build -- otherwise there would be no reason for the build.

For HTTP/2-friendly builds, we’re generating many smaller files; we only want to increment their version when something has changed.

For example, imagine a build that generates vendor-v1.js and application-v1.js; it also generates a scout that loads these two files. We then make a change to an application file, and we do another build, creating vendor-v2.js and application-v2.js. However, no vendor files have changed; our scout should now load to application-v2.js but still load vendor-v1.js. If our scout points to vendor-v2.js, we lose the benefit of being able to cache smaller pieces of our code.

This can be solved by using hashes of the file contents rather than version numbers: vendor-d41d8cd98f.js. If a file has not changed, its hash will remain the same. (Notably, inconsequential changes will change the hash -- for example, a new copyright comment that is inserted post-minification.) Plenty of build strategies already use content hashes for versioning; however, many still use integers, dates, or commit hashes, which change even when the contents of a file have not.

Given files whose names include a hash, our scout can include a manifest that prescribes the file to load for a given resource. The manifest would be generated by the build after all of the resources were generated.

module.exports = {
  baseUrl : 'https://mysite.com/static/',
  resources : {
    vendor : 'vendor-d41d8cd98f.js',
    application : 'application-a32e3ec23d.js'
  }
};

Push: Because you downloaded scout.js, you might also like ... #

Another exciting opportunity in an HTTP/2 world is the ability to push a cascade of resources.

The first push opportunity is the scout itself: for sites and applications that currently ship the scout inlined in the initial HTML payload, server push affords an opportunity to send the scout as a separate resource when the initial HTML is requested.

There’s an interesting dilemma here: If the browser already has the resource cached, and the cache is still valid, it doesn’t need the server to push the resource. Currently, though, there’s no way for the browser to communicate its cache contents to the server. A browser can decline a push, but the server may have already started to send it. We’ve basically introduced a new tradeoff: server push can get the resource to the browser quickly, but we waste bandwidth if the browser doesn’t need it.

As discussed at the link above, a smart server could use session information to determine when to push -- for example, if the page is reloaded within a resource’s cache time, there is no need to re-push that resource to the same session -- but this makes push state-dependent, a frightening prospect if we hope to use CDNs to ensure efficient asset delivery.

Assuming we've generated a manifest as described above, we have the option of going a step further: we can separate the manifest and the scout, allowing the scout to have a far longer cache time than in a pre-HTTP/2 world. This is possible because the thing that typically changes about the scout is the version of the resources it loads, and it makes the most sense on a site where there are different payloads for different pages or users. For applications that previously included the scout in HTML, we can push the scout and the manifest, and have the scout request the manifest; for applications that loaded the scout as its own JS file, we can push the manifest when the scout file is loaded and, again, have the scout request the manifest.

This approach also makes a further case for a standardized scout: application-specific configuration can be shipped in the manifest, and a standardized scout can be shared across applications. This scout could be a file loaded via a script tag, where the script tag itself provides information about the application manifest to use:

<script src="/static/shared/js/scout.js"
  data-manifest="/static/apps/myapp/manifest.js"></script>

The manifest contains information about the other resources that the scout will request, and can even be used by the server to determine what to push alongside the HTML.

A manifest could provide these instructions:

module.exports = {
  baseUrl : 'https://mysite.com/static/',
  resources : {
    vendor : {
      version : 'vendor-d41d8cd98f.js',
      pushWith : [ 'scout' ]
    },
    application : {
      version : 'application-a32e3ec23d.js',
      pushWith : [ 'scout' ]
    },
    secondary : {
      version : 'secondary-e43b8ad12f.js',
      pushWith : [ ]
    }
  }
};

Processing this manifest would require intelligence on the part of the CDN; it may be necessary to replace s3 storage with an actual server that is capable of making these decisions, fronted by a CDN that can intelligently relay responses that include server push.

The elephants in the room #

There are two notable challenges to the rapid transition to an HTTP/2 world: the continued existence of legacy browsers, especially on mobile; and the requirement that HTTP/2 connections be conducted over TLS. Thankfully, the latter provides a reasonable opportunity to address the former. Let's, then, talk about the TLS requirement first.

HTTP/2 is a new protocol, and as such, it is greatly confusing to a large segment of the existing internet: proxies, antivirus software, and the like. During the development of HTTP/2 and SPDY before it, engineers observed that traffic that was transported on an insecure connection would frequently fail. The reason? The proxies, the antivirus software, and all the rest had certain expectations of HTTP traffic; HTTP/2 violated those expectations, and so HTTP/2 traffic was considered unsafe. The software that thwarted insecure HTTP/2 traffic didn't have the ability to inspect secure traffic, and so HTTP/2 traffic over a secure connection passed through just fine. Thus was born the requirement — which is a browser implementation detail, and not part of the HTTP/2 spec — that HTTP/2 web communication be conducted using TLS.

The Let's Encrypt project aims to eliminate the high cost of obtaining the certificate that enables secure HTTP communication; there will still be technical hurdles to using that certificate, but those should be surmountable for anyone who cares enough to engineer a performant HTTP/2 deployment.

In order for a browser and a server to communicate using HTTP/2, the browser and the server must first agree that they can. The TLS handshake that enables secure communication turns out to be the ideal time to negotiate the communication protocol, as well: no additional round trip is required for the negotiation.

When a server is handling a request, it knows whether the browser understands HTTP/2; we can use this information to shape our payload. We can send a legacy browser an HTML file that includes an inlined scout file, and that inlined scout file can include the manifest. The manifest can provide information about how to support legacy browsers:

module.exports = {
  baseUrl : 'https://mysite.com/static/',
  resources : {
    // ...
  },
  legacyResources : {
    legacyMain : {
      initialLoad : true,
      version : 'legacy-main-c312efa43e.js'
    },
    legacySecondary : {
      version : 'legacy-secondary-a22cf1e2af.js'
    }
  }
};

For Consideration: HTTP/2-friendly deployments with HTTP/1.1 support #

Putting the pieces together, we arrive at a deployment process that does the following:

Generates files that contain one or more modules, grouped by likelihood of changing, functionality, or another strategy. The file grouping strategy must persist across builds; new groupings would need a new, unique name that had not been used by earlier builds.
Generates legacy files, where those files contain modules that are grouped according to their likelihood to change, and according to whether they are required for initial load.
Names all files with a content hash.
Generates a manifest for the build, where the manifest includes:
- a baseUrl property whose value is a string that should be used as the base for generating a full URL to a resource, using the pattern <baseUrl>/<resource.version>
- a resources property whose value is an object that, for each file, provides:
  - the most recent changed version
  - a list of individual files which, when any of the files is requested, should trigger a push of the bundle
- a legacyResources property whose value is an object that, for each legacy bundle, provices:
  - the most recent changed version
  - an optional initialLoad property whose value is true if the resource should be loaded immediately by the scout
Generates an HTTP/2 scout file* that provides the ability to load resources, and that loads a manifest.
Generates an HTTP/1 scout file* that provides the ability to load resources, and that includes the manifest.
Uploads the static resources.
Updates a delivery mechanism (such as a server or a CDN) based on the data in the new manifest.

The versioning and caching of the resources would be as follows:

manifest Unversioned. Short cache time, e.g. 10 minutes, to allow for the rapid uptake of new resources for HTTP/2 browsers.
scout Unversioned. Medium cache time, e.g. one day, assuming the contents of this file are considered relatively stable.
legacy-scout Unversioned. Short cache time, e.g. 10 minutes, to allow for the rapid uptake of new resources for legacy browsers.
application and vendor files Versioned. Long cache time, e.g. one year, given that new versions will be picked up when a new manifest is loaded.

* In applications that a) control the initial HTML payload, and b) only use the scout to load other resources, it may not make sense to have a separate scout; it might be sufficient to just load those resources via <script> and <link> tags in the HTML itself. This approach isn't viable for applications that do not control the initial HTML payload, such as third-party applications.

Reality check #

In several places so far, I’ve talked about the need for a server to make decisions about which resources it delivers, and when and how it delivers them. As I alluded to earlier, this could be profoundly challenging for CDNs, which traditionally simply receive a request and return a single resource in response. It also suggests the need for close collaboration between client and server development teams, and an increased knowledge of server-side technology for client-side developers.

CDN support of HTTP/2 in general is rather disappointing, with some major vendors providing nothing more than vague timelines for non-specific support.

As of this writing, I'm unaware of CDNs that support any notion of server push, but I'd be happy to find I am ill-informed. Ideally, CDNs need to provide applications with the ability to express how static assets relate to each other -- a task complicated by the fact that those relationships may be situational, such as in the case where an application doesn't want to push an asset that was just pushed to the same client 10 seconds before. One-size-fits-all push could be accomplished by setting a header on a file, indicating that other files should be pushed alongside it, but that doesn't allow for expressing more nuanced rules.

Even for applications that just want to split their payload into smaller files to take advantage of HTTP/2, and that don't intend to use server push, there is still a gap when it comes to providing a positive experience for HTTP/1.1 clients. CDNs need to surface the ability to change a response not just based on the URL that is requested, but the protocol of the request. Without this ability, we'll be stuck having to choose which protocol to support.

There is also work to be done on tooling, especially if we want to support HTTP/2 without significantly degrading the experience for legacy browsers. Ideally, our build tooling would figure out the optimal combination of files for us, with a knowledge of how the application was bundled previously so as not to squander past caching.

The developer story for HTTP/2 also leaves a lot to be desired as of this writing. Front-end developers are among the most likely in an organization to advocate for this new technology, but my experiences over a few weeks of learning about HTTP/2 suggest that the effort required to set up even a local environment will stretch the comfort zone for many. With a working local environment in hand, the tools to understand the differences between HTTP/2 and HTTP/1 behavior are limited and often confusing. Chrome presents information in its network tab that seems to conflict with the wall of text in its net-internals tool, especially when it comes to server push . Charles Proxy doesn't yet speak HTTP/2. Firefox shows pushed resources as an entry in the network tab, but they appear as though they were never received. nghttp2 provides great insight into how an HTTP/2 server is behaving, but it doesn't speak HTTP/1.1, so you can't use it to do comparisons. Measuring performance using a tool like WebPagetest requires a real certificate, which you may not have handy if you're just trying to experiment.

Alex wrote his 2013 post to document the product of years of experience in creating performant HTTP/1.1 deployments. HTTP/2 means we need to rethink everything we know about shipping applications to the web, and while the building blocks are there, there's still much to figure out about how we'll use them; the "right" answers are, in many cases, still TBD while we wait for vendors to act.

Thanks #

Thanks to the many folks who have talked to me about the ideas in this post, but especially to Lon Ingram, Jake Archibald, and Andy Davies.

Five questions

2015-10-08T00:00:00Z

I recently started in a new role: I'm the dev lead of a project that was already in the hands of a group of skilled developers before I showed up, a project whose scope and technologies extend far beyond the experiences I've had up until now.

As you might imagine, there have been a lot of challenges, but one that's been particularly interesting has been figuring out how to meaningfully contribute to decisions about systems I don't intimately understand. It's easy to be inclined to sit those conversations out: I really don't yet know enough to participate, and the "who am I to have a say?" instinct is strong.

The problem: that attitude will ensure that I never know enough to participate, and though I am definitely out of my comfort zone, my job -- the job I asked to do, and the job I have been asked to do -- is to participate, to learn, and to change the definition of my comfort zone.

While I may not have the project-specific experience to lean on, I'm finding that there are a few questions that help me understand, discuss, and -- ultimately -- consent or object to a technical plan. They're questions that seem to work well across a spectrum of discussions; they work whether we're talking about a wholly new system, a migration from an old system, or a solution to a particularly prickly problem.

These questions don't just help me gain a better understanding of a topic, or help me make better decisions; they've also helped me reframe my understanding of my role as a lead.

Question 1: What are we doing and why? #

When I hear the answer, I'm listening for whether the developer is clearly articulating the problem and the solution. Do we clearly understand the problem? Is the solution magical, or can we explain why it works? Are we solving more than the problem, and thereby incurring unnecessary risk? Does the developer agree that the work is necessary?

Question 2: How could it go wrong? #

A developer who says nothing can go wrong probably hasn't been a developer for very long. I want to hear far-fetched scenarios, and an explanation for why they're far-fetched. I want to hear worst-case scenarios; good developers have already thought about these plenty, they've worked to avoid them, and yet they acknowledge their existence. The goal of this question isn't to plan for everything; rather, the answers provide context for poking at assumptions.

Question 3: How will we know if it’s going wrong? #

This is probably my favorite question. If we're talking about developing a new system or project, it's a question of how we'll know we're off track, which leads to clear milestones and check-in points. If it's a migration to a new system, or a solution to a bad bug, it's a question of how we'll know that the new state is less good than we thought it would be. If the answer is "customers will tell us," we're in dangerous territory. For services, I hope to hear answers about automated monitoring, but manual checks will suffice. For new projects, I hope to hear near-term goals that will help us gauge progress.

Question 4: What will we do if it goes wrong? #

The answer to this may not always be knowable -- obviously we won't always know the ways things will go wrong -- but it's a useful exercise nonetheless. The answer may be "we'll have to revert back to the old system and that will be very hard," but that at least helps me understand the stakes of the decision. For new projects, this is a great way to identify the point of no return -- that is, the point in the project where starting over or changing course
becomes prohibitive.

Question 5: Is there an “undo” button? #

Sometimes, the worst happens. Do we have an escape hatch? How hard will it be to add one later vs. adding one now? Again, it may be OK if we don't have a rollback plan, but knowing that answer should help guide the decision about whether to proceed.

I'm learning that a lot of what makes me kind of OK (I hope!) at this dev lead thing isn't a deep knowledge of the specific technologies that are the underpinning of the project (though it's certainly important that I be able to find my way around). Rather, it's my ability to ask these questions, and to hear and understand the answers, and interpret them into action. I'm thankful to the team that is giving me the chance.

A Baseline for Front-End [JS] Developers, 2015

2015-03-23T00:00:00Z

This post is really old! I've kept it around because it may still be interesting, but many things may be out of date.

It's been almost three years since I wrote A Baseline for Front-End Developers, probably my most popular post ever. Three years later, I still get Twitter mentions from people who are discovering it for the first time.

In some ways, my words have aged well: there is, shockingly, nothing from that 2012 post that has me hanging my head in shame. Still, though: three years is a long time, and a whole lot has changed. In 2012 I encouraged people to learn browser dev tools and get on the module bandwagon; CSS pre-processors and client-side templating were still worthy of mention as new-ish things that people might not be sold on; and JSHint was a welcome relief from the #getoffmylawn admonitions -- accurate though they may have been -- of JSLint.

It's 2015. I want to write an update, but as I sit down to do just that, I realize a couple of things. One, it's arguably not fair to call this stuff a "baseline" -- if you thought that about the original post, you'll find it doubly true for this one. One could argue we should consider the good-enough-to-get-a-job skills to be the "baseline." But there are a whole lot of front-end jobs to choose from, and getting one doesn't establish much of a baseline. For me, I don't want to get a job; I want to get invited to great jobs. I don't want to go to work; I want to go to work with talented people. And I don't want to be satisfied with knowing enough to do the work that needed to be done yesterday; I want to know how to do the work that will need to get done tomorrow.

Two, my world has become entirely JavaScript-centric: knowledge of the ins and outs of CSS has become less and less relevant to my day-to-day work, except where performance is concerned. I know there are plenty of very smart front-end developers for whom this isn't true, but I have also noticed a growing gulf between those who focus on CSS and those who focus on JavaScript. That's probably a subject for another blog post, but I bring it up just to say: I am woefully unequipped to make recommendations about what you should know about CSS these days, so I'm not going to try.

In short: if this list of things doesn't fit your vision of the front-end world, that's OK! We're both still good people. Promise.

JavaScript #

Remember back in 2009 when you read that HTML5 would be ready to use in 2014, and that seemed like a day that would never come? If so, you're well prepared for the slow-but-steady emergence of ES6 (which is now called ES2015, a name that is sure to catch on any day now), the next version of JavaScript. Getting my bearings with ES6 -- er, ES2015 -- is hands-down my biggest JavaScript to-do item at the moment; it is going to be somewhere between game-changing and life-altering, what with classes, real privacy, better functions and arguments, import-able modules, and so much more. Those who are competent and productive with the new syntax will have no trouble standing out in the JS community. Required reading:

Understanding ES6, a work-in-progress book being developed in the open by Nicholas Zakas.
BabelJS, a tool that lets you write ES6 today and "compile" it to ES5 that will run in current browsers. They also have a good learning section.
ES6 Rocks, with various posts that explore ES6 features, semantics, and gotchas.

Do you need to be an ES6/ES2015 expert? Probably not today, but you should know at least as much about it as your peers, and possibly more. It's also worth at least entertaining the possibility of writing your next greenfield project using ES6; the future will be here before you know it.

New language features aside, you should be able to speak fluently about the asynchronicity of JavaScript, and using callbacks and promises to manage it. You should have well-formed opinions about strategies for loading applications in the browser and communicating between pieces of an application. You should maybe have a favorite application development framework, but not at the expense of having a general understanding of how other frameworks operate, and the tradeoffs you accept when you choose one.

Modules & Build Tools #

There's no debate that modules should be the building blocks of client-side web applications. Back in 2012, there was lots of debate about what kind of modules we should use for building apps destined for the browser -- AMD or CommonJS. The somewhat-gross UMD wrapper arose to try to avoid answering the question while still allowing code reuse -- because hey, what's a few more bytes between friends?

I don't feel like this debate is anywhere near resolved, but this is the area where I feel like we've seen the largest transformation since my 2012 article, though perhaps that's a reflection of my personal change of heart. I'm not ready to say that I'm done with AMD, but let's just say I'm floored by how practical it has become to develop and deploy web applications using CommonJS, including modules imported with npm.

With much love for all that RequireJS has contributed to the module conversation, I'm a bit enamored of webpack right now. Its features -- such as easy-to-understand build flags -- feel more accessible than RequireJS. Its hot-swap builds via its built-in dev server make for a fast and delightful development story. It doesn't force an AMD vs. CommonJS decision, because it supports both. It also comes with a ton of loaders, making it fairly trivial to do lots of common tasks. Browserify is worth knowing about, but lags far behind Webpack in my opinion. Smart people I trust tell me that systemjs is also a serious contender in this space, but I haven't used it yet, and its docs leave me wanting. Its companion package manager jspm is intriguing, allowing you to pull in modules from multiple sources including npm, but I'm a bit wary of combining those two concerns. Then again, I never thought I'd break up with AMD, yet here I seem to be, so we'll see.

I still long for a day when we stop having module and build tool debates, and there is a single module system and sharing code between arbitrary projects becomes realistic and trivial without the overhead of UMD. Ideally, the arrival of ES6 modules will bring that day -- and transpilers will fill in the gaps as the day draws closer -- but I find it just as likely that we'll keep finding ways to make it complicated.

In the meantime, front-end developers need to have an opinion about at least a couple of build tools and the associated module system, and that opinion should be backed up by experience. For better or worse, JavaScript is still in a state where the module decision you make will inform the rest of your project.

Testing #

Testing of client-side code has become more commonplace, and a few new testing frameworks have arrived on the scene, including Karma and Intern. I find Intern's promise-based approach to async testing to be particulary pleasing, though I confess that I still write most of my tests using Mocha -- sometimes I'm just a creature of habit.

The main blocker to testing is the code that front-end devs tend to write. I gave a talk toward the end of 2012 about writing testable JavaScript, and followed up with an article on the topic a few months later.

The second biggest blocker to testing remains the tooling. Webdriver is still a huge pain to work with. Continuous automated testing of a complex UI across all supported browsers continues to be either impossible, or so practically expensive that it might as well be impossible -- and never mind mobile. We're still largely stuck doing lightweight automated functional tests on a small subset of supported browser/device/OS combinations, and leaning as hard as we can on lower-level tests that can run quickly and inexpensively. This is a bummer.

If you're interested in improving the problem of untested -- or untestable -- code, the single most valuable book you can read is Working Effectively with Legacy Code. The author, Michael Feathers, defines "legacy code" as any code that does not have tests. On the topic of testing, the baseline is to accept the truth of that statement, even if other constraints are preventing you from addressing it.

Process Automation #

You, hopefully, take for granted the existence of Grunt for task automation. Gulp and Broccoli provide a different approach to automating builds in particular. I haven't used Broccoli, and I've only dabbled in Gulp, but I've definitely come to appreciate some of the limitations of Grunt when it comes to automating complex tasks that depend on other services -- especially when that task needs to run thousands of times a day.

The arrival of Yeoman was a mere 45 days away when I wrote my 2012 post. I confess I didn't use it when it first came out, but recently I've been a) starting projects from scratch using unfamiliar tech; and b) trying to figure out how to standardize our approach to developing third-party JS apps at Bazaarvoice. Yeoman really shines in both of these cases. A simple yo react-webpack from the command line creates a whole new project for you, with all the bells and whistles you could possibly want -- tests, a dev server, a hello world app, and more. If React and Webpack aren't your thing, there's probably a generator to meet your needs, and it's also easy to create your own.

Given that Yeoman is a tool that you generally use only at the start of a project, and given that new projects don't get started all the time, it's mostly just something worth knowing about. Unless, of course, you're also trying to standardize practices across projects -- then it might be a bit more valuable.

Broccoli has gotten its biggest adoption as the basis for ember-cli, and folks I trust suggest that pairing may get a makeover -- and a new name -- to form the basis of a Grunt/Yeoman replacement in the future. Development on both Grunt and Yeoman has certainly slowed down, so it will be interesting to see what the future brings there.

Code Quality #

If you, like me, start to twitch when you see code that violates a project's well-documented style guide, then tools like JSCS and ESLint are godsends, and neither of them existed for you to know about them back in 2012. They both provide a means to document your style guide rules, and then verify your code against those rules automatically, before it ever makes it into a pull request. Which brings me to …

Git #

I don't think a whole lot has changed in the world of Git workflows since 2012, and I'd like to point out Github still hasn't made branch names linkable on the pull request page, for f@#$s sake.

You should obviously be comfortable working with feature branches, rebasing your work on the work of others, squashing commits using interactive rebase, and doing work in small units that are unlikely to cause conflicts whenever possible. Another Git tool to add to your toolbox if you haven't already is the ability to run hooks -- specifically, pre-push and pre-commit hooks to run your tests and execute any code quality checks. You can write them yourself, but tools like ghooks make it so trivial that there's little excuse not to integrate them into your workflow.

Client-Side Templating #

This may be the thing I got the most wrong in my original post, for some definition of "wrong." Client-side templating is still highly valuable, of course -- so valuable that it will be built-in to ES2015 -- but there can be too much of a good thing. It's been a hard-earned lesson for lots of teams that moving all rendering to the browser has high costs when it comes to performance, and thus has the "generate all the HTML client-side" approach rightfully fallen out of favor. Smart projects are now generating HTML server-side -- maybe even pre-generating it, and storing it as static files that can be served quickly -- and then "hydrating" that HTML client-side, updating it with client-side templates as events warrant.

The new expectation here -- and I say this to myself as much as to anyone else -- is that you are considering the performance implications of your decisions, and maybe not restricting yourself quite so thoroughly to the realm of the browser. Which, conveniently, leads to …

Node #

You say you know JavaScript, so these days I expect that you can hop on over to the Node side of things and at least pitch in, if not get at least knee-deep. Yes, there are file systems and streams and servers -- and some paradigms that are fundamentally different from front-end dev -- but front-end developers who keep the back end at arm's length are definitely limiting their potential.

Even if your actual production back-end doesn't use Node, it's an invaluable tool when it comes to keeping you from getting blocked by back-end development. At the very least, you should be familiar with how to initialize a Node project; how to set up an Express server and routes; and how use the request module to proxy requests.

The End #

Thanks to Paul, Alex, Adam, and Ralph for their thorough review of this post, and for generously pointing out places where I could do better. Thank them for the good parts, and blame any errors on me.

With that, good luck. See you again in another three years, perhaps.

Writing Unit Tests for Existing JavaScript

2014-07-13T00:00:00Z

This post is really old! I've kept it around because it may still be interesting, but many things may be out of date.

My team at Bazaarvoice has been spending a lot of time lately thinking about quality and how we can have greater confidence that our software is working as it should.

We've long had functional tests in place that attempt to ask questions like "When a user clicks a button, will The Widget do The Thing?" These tests tell us a fair amount about the state of our product, but we've found that they're brittle -- even after we abstracted away the CSS selectors that they rely on -- and that they take approximately forever to run, especially if we want to run them in all of the browsers we support. The quality of the tests themselves is all over the map, too -- some of them are in fact unit tests, not really testing anything functional at all.

A few months ago we welcomed a new QA lead to our team as part of our renewed focus on quality. Having a team member who is taking a thoughtful, systematic approach to quality is a game-changer -- he's not just making sure that new features work, but rather has scrutinized our entire approach to delivering quality software, to great effect.

One of the things he has repeatedly emphasized is the need to push our tests down the stack. Our functional tests should be black-box -- writing them shouldn't require detailed knowledge of how the software works under the hood. Our unit tests, on the other hand, should provide broad and detailed coverage of the actual code base. In an ideal world, functional tests can be few and slow-ish, because they serve as an infrequent smoke test of the application; unit tests should be thorough, but execute quickly enough that we run them all the time.

Until now, our unit tests have been entirely focused on utility and framework code -- do we properly parse a URL, for example? -- not on code that's up close and personal with getting The Widget to do The Thing. I'd told myself that this was fine and right and good, but in reality I was pretty terrified of trying to bolt unit tests onto feature code of incredibly varying quality, months or even years after it was first written.

A week or so ago, thanks to some coaxing/chiding from fellow team members, I decided to bite the bullet and see just how bad it would be. A week later, I feel like I've taken the first ten steps in a marathon. Of course, taking those first steps involves making the decision to run, and doing enough training ahead of time that you don't die, so in that regard I've come a long way already. Here's what I've done and learned so far.

Step 0 #

I was lucky in that I wasn't starting entirely from scratch, but if you don't already have a unit testing framework in place, don't fret -- it's pretty easy to set up. We use Grunt with Mocha as our test framework and expect.js as our assertion library, but if I were starting over today I'd take a pretty serious look at Intern.

Our unit tests are organized into suites. Each suite consists of a number of files, each of which tests a single AMD module. Most of the modules under test when I started down this path were pretty isolated -- they didn't have a ton of dependencies generally, and had very few runtime dependencies. They didn't interact with other modules that much. Almost all of the existing unit test files loaded a module, executed its methods, and inspected the return value. No big deal.

Feature-related code -- especially already-written feature-related code -- is a different story. Views have templates. Models expect data. Models pass information to views, and views pass information to models. Some models need parents; others expect children. And pretty much everything depended on a global-ish message broker to pass information around.

Since the code was originally written without tests, we are guaranteed that it would be in various states of testability, but a broad rewrite for testability is of course off the table. We'll rewrite targeted pieces, but doing so comes with great risk. For the most part, our goal will be to write tests for what we have, then refactor cautiously once tests are in place.

We decided that the first place to start was with models, so I found the simplest model I could:

define([
  'framework/bmodel',
  'underscore'
], function (BModel, _) {
  return BModel.extend({
    options : {},
    name : 'mediaViewer',

    init : function (config, options) {
      _.extend(this.options, options);
    }
  });
});

Why do we have a model that does approximately nothing? I'm not going to attempt to answer that, though there are Reasons -- but for the sake of this discussion, it certainly provides an easy place to start.

I created a new suite for model tests, and added a file to the suite to test the model. I could tell you that I naively plowed ahead thinking that I could just load the module and write some assertions, but that would be a lie.

Mocking: Squire.js #

I knew from writing other tests, on this project and projects in the past, that I was going to need to "mock" some of my dependencies. For example, we have a module called ENV that is used for ... well, way too much, though it's better than it used to be. A large portion of ENV isn't used by any given module, but ENV itself is required by essentially every model and view.

Squire.js is a really fantastic library for doing mocking in RequireJS-land. It lets you override how a certain dependency will be fulfilled; so, when a module under test asks for 'ENV', you can use Squire to say "use this object that I've hand-crafted for this specific test instead."

I created an Injector module that does the work of loading Squire, plus mocking a couple of things that will be missing when the tests are executed in Node-land.

define([
  'squire',
  'jquery'
], function (Squire, $) {
  return function () {
    var injector;

    if (typeof window === 'undefined') {
      injector = new Squire('_BV');

      injector.mock('jquery', function () {
        return $;
      });

      injector.mock('window', function () {
        return {};
      });
    }
    else {
      injector = new Squire();
    }

    return injector;
  };
});

Next, I wired up the test to see how far I could get without mocking anything. Note that the main module doesn't actually load the thing we're going to test -- first, it sets up the mocks by calling the injector function, and then it uses the created injector to require the module we want to test. Just like a normal require, the injector.require is async, so we have to let our test framework know to wait until it's loaded before proceeding with our assertions.

define([
  'test/unit/injector'
], function (injector) {
  injector = injector();

  var MediaViewer;

  describe('MediaViewer Model', function () {
    before(function (done) {
      injector.require([
        'bv/c2013/model/mediaViewer'
      ], function (M) {
        MediaViewer = M;
        done();
      });
    });

    it('should be named', function () {
      var m = new MediaViewer({});
      expect(m.name).to.equal('mediaViewer');
    });

    it('should mix in provided options', function () {
      var m = new MediaViewer({}, { foo : 'bar' });
      expect(m.options.foo).to.equal('bar');
    });
  });
});

This, of course, still failed pretty spectacularly. In real life, a model gets instantiated with a component, and a model also expects to have access to an ENV that has knowledge of the component. Creating a "real" component and letting the "real" ENV know about it would be an exercise in inventing the universe, and this is exactly what mocks are for.

While the "real" ENV is a Backbone model that is instantiated using customer-specific configuration data, a much simpler ENV suffices for the sake of testing a model's functionality:

define([
  'backbone'
], function (Backbone) {
  return function (injector, opts) {
    injector.mock('ENV', function () {
      var ENV = new Backbone.Model({
        componentManager : {
          find : function () {
            return opts.component;
          }
        }
      });

      return ENV;
    });

    return injector;
  };
});

Likewise, a "real" component is complicated and difficult to create, but the pieces of a component that this model needs to function are limited. Here's what the component mock ended up looking like:

define([
  'underscore'
], function (_) {
  return function (settings) {
    settings = settings || {};

    settings.features = settings.features || [];

    return {
      trigger : function () {},
      hasFeature : function (refName, featureName) {
        return _.contains(settings.features, featureName);
      },
      getScope : function () {
        return 'scope';
      },
      contentType : settings.contentType,
      componentId : settings.id,
      views : {}
    };
  };
});

In the case of both mocks, we've taken some dramatic shortcuts: the real hasFeature method of a component is a lot more complicated, but in the component mock we create a hasFeature method whose return value can be easily known by the test that uses the mock. Likewise, the behavior of the componentManager's find method is complex in real life, but in our mock, the method just returns the same thing all the time. Our mocks are designed to be configurable by -- and predictable for -- the test that uses it.

Knowing what to mock and when and how is a learned skill. It's entirely possible to mock something in such a way that a unit test passes but the actual functionality is broken. We actually have pretty decent tests around our real component code, but not so much around our real ENV code. We should probably fix that, and then I can feel better about mocking ENV as needed.

So far, my approach has been: try to make a test pass without mocking anything, and then mock as little as possible after that. I've also made a point of trying to centralize our mocks in a single place, so we aren't reinventing the wheel for every test.

Finally: when I first set up the injector module, I accidentally made it so that the same injector would be shared by any test that included the module. This is bad, because you end up sharing mocks across tests -- violating the "only mock what you must" rule. The injector module shown above is correct in that it returns a function that can be used to create a new injector, rather than the injector itself.

Here's what the final MediaViewer test ended up looking like:

define([
  // This properly sets up Squire and mocks window and jQuery
  // if necessary (for running tests from the command line).
  'test/unit/injector',

  // This is a function that mocks the ENV module.
  'test/unit/mocks/ENV',

  // This is a function that mocks a component.
  'test/unit/mocks/component'
], function (injector, ENVMock, component) {
  injector = injector();

  // This will become the constructor for the model under test.
  var MediaViewer;

  // Create an object that can serve as a model's component.
  var c = component();

  // We also need to mock the ENV module and make it aware of
  // the fake component we just created.
  ENVMock(injector, { component : c });

  describe('MediaViewer Model', function () {
    before(function (done) {
      injector.require([
        'bv/c2013/model/mediaViewer'
      ], function (M) {
        MediaViewer = M;
        done();
      });
    });

    it('should be named', function () {
      var m = new MediaViewer({
        component : c
      }, {});
      expect(m.name).to.equal('mediaViewer');
    });

    it('should mix in provided options', function () {
      var m = new MediaViewer({
        component : c
      }, { foo : 'bar' });

      expect(m.options.foo).to.equal('bar');
    });
  });
});

Spying: Sinon #

After my stunning success with writing 49 lines of test code to test a 13-line model, I was feeling optimistic about testing views, too. I decided to tackle this fairly simple view first:

define([
  'framework/bview',
  'underscore',
  'hbs!contentAuthorProfileInline',
  'mf!bv/c2013/messages/avatar',
  'bv/util/productInfo',
  'framework/util/bvtracker',
  'util/specialKeys'
], function (BView, _, template, msgPack, ProductInfo, BVTracker, specialKeys) {
  return BView.extend({
    name : 'inlineProfile',

    templateName : 'contentAuthorProfileInline',

    events : {
      'click .bv-content-author-name .bv-fullprofile-popup-target' : 'launchProfile'
    },

    template : template,

    msgpacks : [msgPack],

    launchProfile : function (e) {
      // use r&r component outlet to trigger full profile popup component event
      this.getTopModel().trigger( 'showfullprofile', this.model.get('Author') );

      BVTracker.feature({
        type : 'Used',
        name : 'Click',
        detail1 : 'ViewProfileButton',
        detail2 : 'AuthorAvatar',
        bvProduct : ProductInfo.getType(this),
        productId : ProductInfo.getId(this)
      });
    }
  });
});

It turned out that I needed to do the same basic mocking for this as I did for the model, but this code presented a couple of interesting things to consider.

First, I wanted to test that this.getTopModel().trigger(...) triggered the proper event, but the getTopModel method was implemented in BView, not the code under test, and without a whole lot of gymnastics, it wasn't going to return an object with a trigger method.

Second, I wanted to know that BVTracker.feature was getting called with the right values, so I needed a way to inspect the object that got passed to it, but without doing something terrible like exposing it globally.

Enter Sinon and its spies. Spies let you observe methods as they are called. You can either let the method still do its thing while watching how it is called, or simply replace the method with a spy.

I solved the first problem by defining my own getTopModel method on the model instance, and having it return an object. I gave that object a trigger method that was actually just a spy -- for the sake of my test, I didn't care what trigger did, only how it was called. Other tests [will eventually] ensure that triggering this event has the desired effect on the targeted model, but for the sake of this test, we don't care.

Here's what the test looks like:

describe('#launchProfile', function () {
  var spy;
  var v;

  before(function () {
    spy = sinon.spy();

    v = new InlineProfile({
      // model and component are defined elsewhere
      component : component,
      model : model
    });

    model.set('Author', 'author');

    v.getTopModel = function () {
      return {
        trigger : spy
      };
    };
  });

  it('should trigger showfullprofile event on top model', function () {
    v.launchProfile();

    expect(spy.lastCall.args[0]).to.equal('showfullprofile');
    expect(spy.lastCall.args[1]).to.equal('author');
  });
});

I solved the second problem -- the need to see what's getting passed to BVTracker.feature -- by creating a BVTracker mock where every method is just a spy:

// This is a mock for BVTracker that can be used by unit tests.
define([
  'underscore'
], function (_) {
  return function (injector, opts) {
    var BVTracker = {};

    injector.mock('framework/util/bvtracker', function () {
      _([
        'error',
        'pageview',
        'feature'
      ]).each(function (event) {
        BVTracker[event] = sinon.spy();
      });
    });

    return BVTracker;
  };
});

My test looked at the BVTracker.feature spy to see what it got when the view's launchProfile method was called:

it('should send a feature analytics event', function () {
  v.launchProfile();

  var evt = BVTracker.feature.lastCall.args[0];

  expect(evt.type).to.equal('Used');
  expect(evt.name).to.equal('Click');
  expect(evt.detail1).to.equal('ViewProfileButton');
  expect(evt.detail2).to.equal('AuthorAvatar');
  expect(evt.bvProduct).to.equal('RatingsAndReviews');
  expect(evt.productId).to.equal('product1');
});

I've barely touched on what you can do with spies, or with Sinon in general. Besides providing simple spy functionality, Sinon delivers a host of functionality that makes tests easier to write -- swaths of which I haven't even begun to explore. One part I have explored is its ability to create fake XHRs and to fake whole servers, allowing you to test how your code behaves when things go wrong on the server. Do yourself a favor and spend some time reading through the excellent docs.

What to test ... and not #

I've written tests now for a tiny handful of models and views. Setting up the mocks was a bit of a hurdle -- and there were plenty of other hurdles that are too specific to our project for me to talk about them in detail -- but overall, the hardest part has been figuring out what, exactly, to test. I crafted the examples above to be pretty straightforward, but reality is a lot more complicated.

Writing tests for existing code requires first understanding the code that's being tested and identifying interesting moments in that code. If there's an operation that affects the "public" experience of the module -- for example, if the value of a model attribute changes -- then we need to write a test that covers that operation's side effect(s). If there's code that runs conditionally, we need to test the behavior of that code when that condition is true -- and when it's not. If there are six possible conditions, we need to test them all. If a model behaves completely differently when it has a parent -- and this happens far too often in our code -- then we need to simulate the parent case, and simulate the standalone case.

It can be tempting to try to test the implementation details of existing code -- and difficult to realize that you're doing it even when you don't mean to. I try to stay focused on testing how other code might consume and interact with the module I'm testing. For example, if the module I'm testing triggers an event in a certain situation, I'm going to write a test that proves it, because some other code is probably expecting that event to get triggered. However, I'm not going to test that a method of a certain name gets called in a certain case -- that's an implementation detail that might change.

The exercise of writing unit tests against existing code proves to be a phenomenal incentive to write better code in the future. One comes to develop a great appreciation of methods that have return values, not side effects. One comes to loathe the person -- often one's past self -- who authored complex, nested conditional logic. One comes to worship small methods that do exactly one thing.

So far, I haven't rewritten any of the code I've been testing, even when I've spotted obvious flaws, and even when rewriting would make the tests themselves easier to write. I don't know how long I'll be able to stick to this; there are some specific views and models that I know will be nearly impossible to test without revisiting their innards. When that becomes necessary, I'm hoping I can do it incrementally, testing as I go -- and that our functional tests will give me the cover I need to know I haven't gone horribly wrong.

Spreading the love #

Our team's next step is to widen the effort to get better unit test coverage of our code. We have something like 100 modules that need testing, and their size and complexity are all over the map. Over the coming weeks, we'll start to divide and conquer.

One thing I've done to try to make the effort easier is to create a scaffolding task using Grunt. Running grunt scaffold-test:model:modelName will generate a basic file that includes mocking that's guaranteed to be needed, as well as the basic instantiation that will be required and a couple of simple tests.

There's another senior team member who has led an effort in the past to apply unit tests to an existing code base, and he's already warned me to expect a bit of a bumpy road as the team struggles through the inevitable early challenges of trying to write unit tests for existing feature code. I expect there to be a pretty steep hill to climb at first, but at the very least, the work I've done so far has -- hopefully -- gotten us to the top of the vertical wall that had been standing in our way.

Show & tell at the Times Open Science Fair

2012-11-25T00:00:00Z

This post is really old! I've kept it around because it may still be interesting, but many things may be out of date.

I spoke at the Times Open Source Science Fair a couple of weeks ago. I'll admit that I was pretty skeptical of the concept when I was first asked, but as someone who used to work as an editor at a tiny newspaper in upstate New York, I wasn't about to say no when the Times asked me to come say hi.

A few days before the event, I got an email asking me for information about what I'd be showing off at my booth. Booth? Wat? They weren't kidding about the science fair thing, but what the heck was I going to show at a booth?

It turns out this is basically the best idea ever. I recruited my Bocoup colleague Rick Waldron to join me, and together we spent a whirlwind hour showing off robots powered by JavaScript to an endless stream of people walking up to our booth. Rick did a great job of setting up a demo that people could play with, and they took turns moving sliding potentiometers that controlled servos that moved an arm with a gripper at the end, trying to pick up Bocoup stickers. Ours was one of about a dozen booths showing off open-source projects, and the room was a wonderful madhouse.

After a break for dinner, I, Jeremy Ashkenas, and Zach Holman each gave 20-minute talks, but the talks were really just icing on the evening. The "science fair" format promoted such intentional interaction, in a way that traditional conferences just can't, no matter how great the hall track or the parties may be. The format invited and encouraged attendees to talk to the presenters -- indeed, if they didn't talk to the presenters, there wasn't much else for them to do. By the time the official talks came around, a super-casual, super-conversational atmosphere had already been established, and the energy that created was tangibly different from any event I've been to before.

I love conferences, and the sharing of knowledge that happens there, and there's a whole lot to be said for their speaker-audience format -- don't get me wrong. But I'd also love to see more events figure out how to integrate this show and tell format. "Booths" don't need to mean "vendors trying to sell things" -- they can actually be a great opportunity to facilitate conversation, and to let open source contributors show off their hard work.

A Baseline for Front-End Developers

2012-04-12T00:00:00Z

This post is really old! I've kept it around because it may still be interesting, but many things may be out of date.

I wrote a README the other day for a project that I'm hoping other developers will look at and learn from, and as I was writing it, I realized that it was the sort of thing that might have intimidated the hell out of me a couple of years ago, what with its casual mentions of Node, npm, Homebrew, git, tests, and development and production builds.

Once upon a time, editing files, testing them locally (as best as we could, anyway), and then FTPing them to the server was the essential workflow of a front-end dev. We measured our mettle based on our ability to wrangle IE6 into submission or achieve pixel perfection across browsers. Many members of the community -- myself included -- lacked traditional programming experience. HTML, CSS, and JavaScript -- usually in the form of jQuery -- were self-taught skills.

Something has changed in the last couple of years. Maybe it's the result of people starting to take front-end dev seriously, maybe it's browser vendors mostly getting their shit together, or maybe it's front-end devs -- again, myself included -- coming to see some well-established light about the process of software development.

Whatever it is, I think we're seeing the emphasis shift from valuing trivia to valuing tools. There's a new set of baseline skills required in order to be successful as a front-end developer, and developers who don't meet this baseline are going to start feeling more and more left behind as those who are sharing their knowledge start to assume that certain things go without saying.

Here are a few things that I want to start expecting people to be familiar with, along with some resources you can use if you feel like you need to get up to speed. (Thanks to Paul Irish, Mike Taylor, Angus Croll, and Vlad Filippov for their contributions.)

JavaScript #

This might go without saying, but simply knowing a JavaScript library isn't sufficient any more. I'm not saying you need to know how to implement all the features of a library in plain JavaScript, but you should know when a library is actually required, and be capable of working with plain old JavaScript when it's not.

That means that you've read JavaScript: The Good Parts -- hopefully more than once. You understand data structures like objects and arrays; functions, including how and why you would call and apply them; working with prototypal inheritance; and managing the asynchronicity of it all.

If your plain JS fu is weak, here are some resources to help you out:

Eloquent Javascript: A wonderful book (also available in print) that takes you back to JavaScript basics
A Test-Driven JS Assessment: A set of failing tests that cover various JavaScript topics; can you write code to make the tests pass?
10 things I learned from the jQuery Source is an oldie but goodie from Paul Irish that shows what you can learn by reading other people's code.

Git (and a Github account) #

If you're not on Github, you're essentially unable to participate in the rich open-source community that has arisen around front-end development technologies. Cloning a repo to try it out should be second-nature to you, and you should understand how to use branches on collaborative projects.

Need to boost your git skills?

Modularity, dependency management, and production builds #

The days of managing dependencies by throwing one more script or style tag on the page are long gone. Even if you haven't been able to incorporate great tools like RequireJS into your workflow at work, you should find time to investigate them in a personal project or in a project like Backbone Boilerplate, because the benefits they convey are huge. RequireJS in particular lets you develop with small, modular JS and CSS files, and then concatenates and minifies them via its optimization tool for production use.

Skeptical of AMD? That's no excuse to be doing nothing. At the very least, you should be aware of tools like UglifyJS or Closure Compiler that will intelligently minify your code, and then concatenate those minified files prior to production.

If you're writing plain CSS -- that is, if you're not using a preprocessor like Sass or Stylus -- RequireJS can help you keep your CSS files modular, too. Use @import statements in a base file to load dependencies for development, and then run the RequireJS optimizer on the base file to create a file built for production.

In-Browser Developer Tools #

Browser-based development tools have improved tremendously over the last couple of years, and they can dramatically improve your development experience if you know how to use them. (Hint: if you're still using alert to debug your code, you're wasting a lot of time.)

You should probably find one browser whose developer tools you primarily use -- I'm partial to Google Chrome's Developer Tools these days -- but don't dismiss the tools in other browsers out of hand, because they are constantly adding useful features based on developer feedback. Opera's Dragonfly in particular has some features that make its developer tools stand out, such as an (experimental) CSS profiler, customizable keyboard shortcuts, remote debugging without requiring a USB connection, and the ability to save and use custom color palettes.

If your understanding of browser dev tools is limited, Fixing these jQuery is a great (and not particularly jQuery-centric) overview of debugging, including how to do step debugging -- a life-altering thing to learn if you don't already know it.

The command line #

Speaking of the command line, being comfortable with it is no longer optional -- you're missing out on way too much if you're not ready to head over to a terminal window and get your hands dirty. I'm not saying you have to do everything in the terminal -- I won't take your git GUI away from you even though I think you'll be better off without it eventually -- but you should absolutely have a terminal window open for whatever project you're working on. There are a few command line tasks you should be able to do without thinking:

ssh to log in to another machine or server
scp to copy files to another machine or server
ack or grep to find files in a project that contain a string or pattern
find to locate files whose names match a given pattern
git to do at least basic things like add, commit, status, and pull
brew to use Homebrew to install packages
npm to install Node packages
gem to install Ruby packages

If there are commands you use frequently, edit your .bashrc or .profile or .zshrc or whatever, and create an alias so you don't have to type as much. You can also add aliases to your ~/.gitconfig file. Gianni Chiappetta's dotfiles are an excellent inspiration for what's possible.

Note: If you're on Windows, I don't begin to know how to help you, aside from suggesting Cygwin. Right or wrong, participating in the open-source front-end developer community is materially more difficult on a Windows machine. On the bright side, MacBook Airs are cheap, powerful, and ridiculously portable, and there's always Ubuntu or another *nix.

Client-side templating #

It wasn't so long ago that it was entirely typical for servers to respond to XHRs with a snippet of HTML, but sometime in the last 12 to 18 months, the front-end dev community saw the light and started demanding pure data from the server instead. Turning that data into HTML ready to be inserted in the DOM can be a messy and unmaintainable process if it's done directly in your code. That's where client-side templating libraries come in: they let you maintain templates that, when mixed with some data, turn into a string of HTML. Need help picking a templating tool? Garann Means' template chooser can point you in the right direction.

CSS preprocessors #

Paul Irish noted the other day that we're starting to see front-end devs write code that's very different from what ends up in production, and code written with CSS preprocessors is a shining example of this. There's still a vocal crowd that feels that pure CSS is the only way to go, but they're starting to come around. These tools give you features that arguably should be in CSS proper by now -- variables, math, logic, mixins -- and they can also help smooth over the CSS property prefix mess.

Testing #

One of the joys of writing modular, loosely coupled code is that your code becomes vastly easier to test, and with tools like Grunt, setting up a project to include tests has never been easier. Grunt comes with QUnit integration, but there are a host of testing frameworks that you can choose from -- Jasmine and Mocha are a couple of my current favorites -- depending on your preferred style and the makeup of the rest of your stack.

While testing is a joy when your code is modular and loosely coupled, testing code that's not well organized can be somewhere between difficult and impossible. On the other hand, forcing yourself to write tests -- perhaps before you even write the code -- will help you organize your thinking and your code. It will also let you refactor your code with greater confidence down the line.

A short screencast I recorded about testing your jQuery with Jasmine.
An example of unit tests on the jquery-bbq plugin.

Process automation (rake/make/grunt/etc.) #

Grunt's ability to set up a project with built-in support for unit tests is one example of process automation. The reality of front-end development is that there's a whole lot of repetitive stuff we have to do, but as a friend once told me, a good developer is a lazy developer: as a rule of thumb, if you find yourself doing the same thing three times, it's time to automate it.

Tools like make have been around for a long time to help us with this, but there's also rake, grunt, and others. Learning a language other than JavaScript can be extremely helpful if you want to automate tasks that deal with the filesystem, as Node's async nature can become a real burden when you're just manipulating files. There are lots of task-specific automation tools, too -- tools for deployment, build generation, code quality assurance, and more.

Code quality #

If you've ever been bitten by a missing semicolon or an extra comma, you know how much time can be lost to subtle flaws in your code. That's why you're running your code through a tool like JSHint, right? It's configurable and has lots of ways to integrate it into your editor or build process.

The fine manual #

Alas, there is no manual for front-end development, but MDN comes pretty close. Good front-end devs know to prefix any search engine query with mdn -- for example, mdn javascript arrays -- in order to avoid the for-profit plague that is w3schools.

The End #

As with anything, reading about these things won't make you an expert, or even moderately skilled -- the only surefire way to get better at a thing is to do that thing. Good luck.

Girls and computers

2012-03-25T00:00:00Z

After a week that seemed just chock full of people being stupid about women in technology, I just found myself thinking back on how it was that I ended up doing this whole computer thing in the first place. I recorded a video a while back for the High Visibility Project, but that really just told the story of how I ended up doing web development. The story of how I got into computers begain when I was unequivocally a girl. It was 1982.

Back then, my dad made eyeglasses. My mom stayed at home with me and my year-old sister -- which she'd continue to do til I was a teenager, when my brother finally entered kindergarten eight years later. Their mortgage was $79 -- about $190 in today's dollars -- which is a good thing because my dad made about $13,000 a year. We lived in Weedsport, New York, a small town just outside of Syracuse. We walked to the post office to get our mail. The farmers who lived just outside town were the rich people. In the winters the fire department filled a small depression behind the elementary school with water for a tiny skating rink. There were dish-to-pass suppers in the gym at church.

In 1982, Timex came out with the Timex Sinclair TS-1000, selling 500,000 of them in just six months. The computer, a few times thicker than the original iPad but with about the same footprint, cost $99.95 -- more than that mortgage payment. When everyone else in town was getting cable, my parents decided that three channels were good enough for them -- it's possible they still had a black-and-white TV -- and bought a computer instead.

I remember tiny snippets of that time -- playing kickball in my best friend Beth's yard, getting in trouble for tricking my mother into giving us milk that we used to make mud pies, throwing sand in the face of my friend Nathan because I didn't yet appreciate that it really sucks to get sand thrown in your face -- but I vividly remember sitting in the living room of our house on Horton Street with my father, playing with the computer.

A cassette player was our disk drive, and we had to set the volume just right in order to read anything off a tape -- there was actually some semblance of a flight simulator program that we'd play, after listening to the tape player screech for minutes on end. Eventually we upgraded the computer with a fist-sized brick of RAM that we plugged into the back of the computer, bumping our total capacity from 2K to 34K. I wrote programs in BASIC, though for the life of me I can't remember what any of them did. The programs that were the most fun, though, were the ones whose assembly I painstakingly transcribed, with my five-year-old fingers, from the back of magazines -- pages and pages of letters and numbers I didn't understand on any level, and yet they made magic happen if I got every single one right.

A string of computers followed. My parents bought a Coleco Adam when we moved to Horseheads, New York -- apparently the computer came with a certificate redeemable for $500 upon my graduation from high school, but Coleco folded long before they could cash it in. I made my first real money by typing a crazy lady's crazy manuscript about crazy food into an Apple IIe that we had plugged into our TV, and my uncle and I spent almost the entirety of his visit from Oklahoma writing a game of Yahtzee! on that computer, again in BASIC.

In middle school, I started a school newspaper, and I think we used some prehistoric version of PageMaker to lay it out. When high school rolled around, I toiled through hand-crafting the perfect letters and lines and arrows in Technical Drawing so I could take CAD and CAM classes and make the computer draw letters and lines and arrows for me, and quickly proceeded to school just about every boy in the class. In my senior year of high school, I oversaw the school yearbook's transition from laying out pages on paper to laying out pages with computers, this time the vaguely portable (it had a handle on the back!) Mac Classic. We used PageMaker again; the screen was black and white and 9", diagonally.

It was around then that a friend gave me a modem and -- to his eventual chagrin, when he got the bill -- access to his Delphi account, giving me my first taste of the whole Internet thing in the form of telnet, gopher, and IRC. When I went to college the following year, I took with me a computer with perhaps a 10MB hard drive, and no mouse.

Once again I found myself poring over magazines to discover URIs and, eventually, URLs that I could type to discover a whole new world of information. In 1995, I spent the summer making my college newspaper's web site, previewing it in Lynx -- it felt like there wasn't much to learn when there was so little difference between the markup and what I saw on the screen. I would go to the computer lab to use NCSA's Mosaic on the powerful RISC 6000 workstations, because they had a mouse. Yahoo! was about one year old. My friend Dave, who lived down the street, installed Windows 95 that summer and invited me over to show me. It was amazing. We were living in the future.

My early years with computers seem pretty tame -- I wasn't tearing them apart or building my own or doing anything particularly interesting with them, but I was using them, I was telling them what to do and they were mostly listening, and it never made me feel like I was weird. To the contrary, it made me feel powerful and empowered. I felt like a part of this ever-growing community of people who understood, eventually, that computers were going to change the world. It was the people who didn't understand this who were weird and beneath us. It was the people who understood computers better than me of whom I stood in awe.

I can barely remember a time when computers weren't a part of my life, and yet when they first entered my life, their presence was incredibly exceptional. These days, of course, computers are ubiquitous, but interaction with them at the copy-assembly-from-the-back-of-a-magazine level is almost nonexistent. Parents who can approach a computer with the same awe and wonder and determination as a child -- as I must imagine that my dad did in 1982 -- are likely equally rare.

In some ways, it is like the very ubiquity of technology has led us back to a world where socially normative gender roles take hold all over again, and the effort we're going to need to put into overcoming that feels overwhelming sometimes. Words can't express my gratitude for the parents I have, for that $99.95 investment they made in me, and for fact that I was lucky enough to be 5 and full of wonder in 1982.

Getting Better at JavaScript

2011-05-20T00:00:00Z

This post is really old! I've kept it around because it may still be interesting, but many things may be out of date.

I seem to be getting a lot of emails these days asking a deceptively simple question: “How do I get better at JavaScript?” What follows are some semi-random thoughts on the subject:

The thing that I’ve come to realize about these questions is that some things just take time. I wish I could write down “Ten Things You Need to Know to Make You Amazing at the JavaScript,” but it doesn’t work that way. Books are fantastic at exposing you to guiding principles and patterns, but if your brain isn’t ready to connect them with real-world problems, it won’t.

The number one thing that will make you better at writing JavaScript is writing JavaScript. It’s OK if you cringe at it six months from now. It’s OK if you know it could be better if you only understood X, Y, or Z a little bit better. Cultivate dissatisfaction , and fear the day when you aren’t disappointed with the code you wrote last month.

Encounters with new concepts are almost always eventually rewarding, but in the short term I’ve found they can be downright demoralizing if you’re not aware of the bigger picture. The first step to being better at a thing is realizing you could be better at that thing, and initially that realization tends to involve being overwhelmed with all you don’t know. The first JSConf , in 2009, was exactly this for me. I showed up eager to learn but feeling pretty cocky about my skills. I left brutally aware of the smallness of my knowledge, and it was a transformational experience: getting good at a thing involves seeking out opportunities to feel small.

One of the most helpful things in my learning has been having access to smart people who are willing to answer my questions and help me when I get stuck. Meeting these people and maintaining relationships with them is hard work, and it generally involves interacting with them in real life, not just on the internet, but the dividends of this investment are unfathomable.

To that end, attend conferences. Talk to the speakers and ask them questions. Write them emails afterwards saying that it was nice to meet them. Subscribe to their blogs. Pay attention to what they’re doing and evangelize their good work.

Remember, too, that local meetups can be good exposure to new ideas too, even if on a smaller scale. The added bonus of local meetups is that the people you’ll meet there are … local! It’s easy to maintain relationships with them and share in learning with them in real life.

(An aside: If your company won’t pay for you to attend any conferences, make clear how short-sighted your company’s decision is and start looking for a new job, because your company does not deserve you. Then, if you can, cough up the money and go anyway. As a self-employed consultant, I still managed to find something like $10,000 to spend on travel- and conference-related expenses last year, and I consider every penny of it to be money spent on being better at what I do. When I hear about big companies that won’t fork over even a fraction of that for an employee who is raising their hand and saying “help me be better at what I do!”, I rage.)

Make a point of following the bug tracker and repository for an active open-source project. Read the bug reports. Try the test cases. Understand the commits. I admit that I have never been able to make myself do this for extended periods of time, but I try to drop in on certain projects now and then because it exposes me to arbitrary code and concepts that I might not otherwise run into.

Read the source for your favorite library, and refer to it when you need to know how a method works. Consult the documentation when there’s some part of the source you don’t understand. When choosing tools and plugins, read the source, and see whether there are things you’d do differently.

Eavesdrop on communities, and participate when you have something helpful to add. Lurk on a mailing list or a forum or in an IRC channel, help other people solve problems. If you’re not a help vampire — if you give more than you take — the “elders” of a community will notice, and you will be rewarded with their willingness to help you when it matters.

Finally, books:

JavaScript: The Good Parts, by Douglas Crockford. It took me more than one try to get through this not-very-thick book, and it is not gospel. However, it is mandatory reading for any serious JavaScript developer.
Eloquent JavaScript , Marijn Haverbeke (also in print). This is another book that I consider mandatory; you may not read straight through it, but you should have it close at hand. I like it so much that I actually bought the print version, and then was lucky enough to get a signed copy from Marijn at JSConf 2011.
JavaScript Patterns, by Stoyan Stefanov. This was the book that showed me there were names for so many patterns that I’d discovered purely through fumbling around with my own code. I read it on the flight to the 2010 Boston jQuery Conference, and it’s definitely the kind of book that I wouldn’t have gotten as much out of a year earlier, when I had a lot less experience with the kinds of problems it addresses.
Object-Oriented JavaScript, by Stoyan Stefanov. It’s been ages since I read this book, and so I confess that I don’t have a strong recollection of it, but it was probably the first book I read that got me thinking about structuring JavaScript code beyond the “get some elements, do something with them” paradigm of jQuery.

Good luck.

Deferreds coming to jQuery 1.5?

2010-12-25T00:00:00Z

This post is really old! I've kept it around because it may still be interesting, but many things may be out of date.

I have updated this post to show the code using the API that was released in jQuery 1.5.0.

A few weeks ago, a major rewrite of jQuery's Ajax functionality landed in the jQuery GitHub repo. Thanks to Julian Aubourg, jQuery looks like it will get a feature that I've desperately wished it had ever since I started spending time with Dojo:

function doAjax(debug) {
  var req = $.ajax({
    url : 'foo.php',
    dataType : 'json',
    success : function(resp) {
      console.log(resp);
    }
  });

  if (debug) {
    req.success(function(resp) {
      console.log("let's see that again!", resp);
    });
  }

  // return the request object so other
  // things can bind to it too!
  return req;
}

doAjax().success(function(resp){
  console.log("Once more, with feeling!", resp);
});

Starting with 1.5 (I'm guessing), users will be able to easily attach callbacks to XHRs ... later! And pass around the return value of $.ajax as an actually useful object with a familiar API, rather than just getting back the native XHR! No longer will we have to bundle up callback functionality -- some of which might be optional, or depend on other code -- inside our success or error callbacks. So hott.

When I heard that these Ajax changes had landed, I got to thinking about how Dojo provides its ability to belatedly attach success and error handlers to its XHRs: underlying its XHR methods is dojo.Deferred. It allows users to assign callback functions for success and error conditions for a task that may not complete immediately. Dojo makes use of this for its XHR stuff, but it's incredibly useful generically, too:

function doSomethingAsync() {
  var dfd = new dojo.Deferred();
  setTimeout(function() {
    dfd.resolve('hello world');
  }, 5000);
  return dfd.promise;
};

doSomethingAsync().then(function(resp) {
  console.log(resp); // logs 'hello world'
});

So, Dojo provided the late callback functionality via deferreds. jQuery now had late callback functionality. Was the deferred functionality hidden in the jQuery Ajax rewrite, waiting to be exposed? Julian and I and several others got to talking in the jQuery development IRC channel, and decided it seemed like an interesting and viable idea. A few days later, Julian's first draft of jQuery.Deferred landed in a branch on GitHub.

It's early days, but there have been a lot of good discussions already about the proposed API and how it should work. Through all of the conversations I've been part of, it's become really clear that no one cares about deferreds until you show them what they actually mean: the ability to register an interest in the outcome of arbitrary asynchronous behavior, even if the outcome has already occurred. Even better, you can register your interest in the outcome of behavior that may or may not be asynchronous.

I assure you that once you have experienced this, you will wonder how you lived without it.

var cache = {};

function doSomethingMaybeAsync(val) {
  if (cache[val]) {
    return cache[val];
  }

  return $.ajax({
    url : 'foo.php',
    data : { value : val },
    dataType : 'json',
    success : function(resp) {
      cache[val] = resp;
    }
  });
}

$.when(doSomethingMaybeAsync('foo'))
  .then(function(resp){
    alert("The value for foo is", resp);
  });

It'll also be possible to do something like you see below. I'm not sure what the exact API will be for creating a generic deferred instance, but I hope it will be something along these lines:

function doIt() {
  var dfd = new $.Deferred();

  setTimeout(function() {
    dfd.resolve('hello world');
  }, 5000);

  return dfd.promise;
}

doIt().then(function(resp) { console.log(resp); }, errorFn);

These changes are sitting in a branch in the jQuery GitHub repo as we speak, and I think it's likely we'll see them move to master sooner than later. It's a nice story of collaboration and community participation that helped make something good -- the Ajax rewrite -- even better.

It's exciting to see jQuery venture a bit more into the abstract. My experiences with Dojo core so far make me think there are probably more opportunities for these sorts of utilities that would be of high value for a substantial number of jQuery users. On the other hand, one of the constant themes of our conversations about deferreds has been the potential for confusion with the new methods. Will the API look familiar and jQuery-like, or will users be confused about the ability to chain methods on something other than a jQuery selection? Are there bigger-picture considerations when it comes to adding new constructors to the jQuery namespace? It'll be interesting to see how these questions sort themselves out, especially if other similar features appear that don't fall neatly under the well-established DOM/Ajax/Events/Effects umbrella.

The conversation's happening on GitHub -- I hope you'll join in.

Thoughts on Working for Yourself

2010-08-14T00:00:00Z

This post is really old! I've kept it around because it may still be interesting, but many things may be out of date.

Two years and three weeks ago, I lost my job. My partner and I had just had an expensive deck built; I’d just bought a gratuitously fancy grill to go on it. My token severance check was in the mail — I’d spend most of it to be able to keep my work computer. And so, six-pack of Yuengling and a pack of Parliament Lights in hand — I hadn’t smoked in years — I sat on said deck near said grill, contemplating a world I’d last contemplated when I, age 6 at the time, came home in the middle of the day to find my just-laid-off dad sitting on the porch.

It was not a good day.

Two years on, things look a lot better than they did. My “salary,” as best as I can figure it, is comfortable. Figuring it is hard: I travel to conferences whenever I want, and work pays for them. I buy new equipment and software when I want, and work pays for it. I don’t work a certain number of hours in a day or a week or a month, so I’ve been able to get my pilot’s license, spend two weeks in Hawaii without drawing on some meticulously calculated vacation balance, and take on projects like TXJS without having to get anyone’s blessing — hard to put a value on that. On the flip side, I’ve had more sleepless nights than I’d dare count, wondering where the money was going to come from next month, wondering whether it was time to walk away from an abusive client with no other work on the horizon, wondering whether my relationship could tolerate all the ups and downs, wondering how it was that little ol' me, the Independent Consultant, had any business telling people what to do.

People ask me, now and then, what advice I have for them about being independent, about working for myself. More than a few of them have suddenly found themselves sitting on that proverbial deck, waiting for that severance check (or not). I’ve written many an email, but I thought I’d take a few minutes to try to assemble all of them into a coherent post.

The Touchy-Feely

Network

Once I decided I was going to stop looking for a job and work for myself, I wrote an email to absolutely everyone I could think of who might help me find work. It felt weird, and it’s also exactly how I landed a job that paid the bills fairly solidly for the next nine months. When you decide to take the plunge, it’s imperative that you let people know. Let them know you’re looking for work, and what kind of work you’re willing to do. Get out and participate in your local tech community, giving talks and connecting with people who can connect you with people. Start writing blog posts, answering questions on forums, helping people in IRC channels. Become known as a helpful and knowledgable person. You do not get to be an introvert. Getting your name out there, when you start and constantly after that, is the No. 1 most valuable thing you can do — at least as important as being good at what you do.

Dive in and Do

I cost a fair bit of money. In return for paying that, clients expect that I can come up to speed on their project quickly and start solving their problems for them — they don’t want to pay for a long ramp-up period before they can even accurately assess whether my time was a good purchase. Part of doing this is having solid fundamental skills — an understanding of basic application development patterns, version control systems, development environments, etc. When I don’t have those fundamental skills for a particular project and I expect that may get in the way of a quick ramp-up, I make that very clear to my clients.

More generally, it’s critical that you demonstrate value as soon as you can, and make sure your clients are well aware of the progress you’re making — if they don’t ask you, tell them anyway. If you get stuck on a particular problem, part of your job is to recognize that you’re stuck, and honestly assess whether this is your own shortcoming or not. If it is, think long and hard about whether you should bill the client for your learning time; if it’s not, at least let the client know about the roadblock and your plan for overcoming it — you never know, the client might re-frame the problem in a way that’s easier to solve.

Don’t Specialize … Yet

When you first start, it pays to be open-minded about the kind of work you’ll take on. It helps get those bills paid, but also, it helps you zero in on exactly how you’re good at providing value to clients. The value you provide may be entirely different than the value you were providing at your job, or even than how you thought you provided value in general. Over time, you can start to market yourself using true stories of how you helped real live clients, and start to develop your niche.

Think Global (and favor clients who do, too)

I have had clients in California, Florida, New York, and places in between, but I’ve never needed to travel to any of them. I find that the work I do is incredibly time and place independent, as long as the client has good systems in place. This means version control, a development environment I can ssh to, liberal use of IM, a sane deployment process (read: not FTP), and some sort of project management tool and/or ticketing system. Projects that have lacked these systems have been more challenging, and these days I tend not to accept them.

Don’t Accept Abuse

There are terrible clients out there, but you do not have to work for them. If a client is impossible despite your best efforts to improve the situation, quit in a professional manner but without remorse. The client is not always right. You will find more work from someone better. Every. Single. Time.

Ignore the Economy

One question that’s come up a lot is how I think the economy has affected my ability to find work. Lehman Brothers would declare bankruptcy less than two months after I lost my job; the Dow would lose 3,000 points in the following weeks. My general theory on the economy question is this: rarely is full-time employment of a web worker an efficient distribution of labor, unless you are working for a very, very large company. The volume of work can fluctuate tremendously. I think of all the hours at previous jobs when there was literally nothing to do, yet the companies kept me around for the moment when there was. This was dumb. I’d like to think that as companies are looking for ways to cut costs, they’ll realize that was dumb, and bring people on as needed. The flip side of that is that those displaced workers are now competing for the consulting work. In the end, I think the economy may be a bit of a wash if you’re good at what you do.

Get Support

This feels like the single most discouraging thing I have to tell people: I am not sure I could have done this without the knowledge that the bills would still get paid if I failed. Our double-income-no-kids salary is embarrassing, but we were very intentional when we bought our house that we wouldn’t take on more than we could afford with one salary, no matter what the lenders told us we could have. My partner’s employer allowed me to enroll in their insurance coverage almost as if I were her spouse (I had to wait a few months for open enrollment, costly months I wouldn’t have had to wait if we were married). The comfort of this knowledge has been amazing, in those first few months when it became clear that I just might not need a job and in those dark months when it’s been unclear where the next check would come from. I am sure working for yourself is possible without these stars aligning, but it would require a braver soul than me.

Practical Matters

Have Clear Payment Terms (and realistic expectations)

I learned early on that I can’t expect anything quicker than Net 30, and that’s a long time — as much as 60 days after I did the work, assuming I bill monthly. One client failed to pay due to some issues with their accounting department, and it got to the point where I had to let them know that I couldn’t continue working with them until I got paid. A deadline was looming; a check arrived FedEx the next day. Be clear about your payment terms. You’ll get Net 15 if you’re lucky; Net 30 is standard. Incentivize them with late penalties if you need, and don’t hesitate to contact a client once that deadline passes.

Decide What You’re Worth

Honestly evaluate what your fee system should be, then stick to it. People who want to pay you less will cause other headaches that will make you wish you’d charged them more without fail. If you enter into any retainer situations, make sure the terms are crystal clear to both sides. Generally retainers work where the client purchases a minimum number of hours per month (potentially in exchange for a bulk discount). Communicate with them if they are not using their minimum hours, but try hard not to end up in a situation where you’ve cleared your plate for a client who then doesn’t need you as much as they thought they would. On the flip side, if the client has promised you 30 hours a week because they overestimated a task, but you’re so good the work is only taking you 15, make that very clear to the client and move toward arriving at a new arrangement quickly. They’ll appreciate your honesty, and you can free up your time for other work.

Decide How Much You Want to Work

Be clear about your availability if it’s not 24/7. Let clients know what the best way is to contact you. For me, for example, I hate being interrupted by phone calls, and greatly prefer IM over email for quick exchanges. I’ve learned to tell my clients this up front. For your own sanity, contemplate whether there’s a minimum number of hours you’re willing to work on a project. Too many times I’ve spent more time discussing a project than actually doing it. Make sure you account for that discussion time, and for the cost of getting you to sit at your computer rather than playing outside.

Get an Accountant

If you find yourself making any money at all, get an accountant. Taxes are complicated for self-employment. April 15 will either suck or it will suck a whole lot. An accountant will help you figure out estimated payments and advise you on the best way to keep as much of your money as you can. For me, that meant forming an LLC, which meant a lot more paperwork throughout the year, but a lot more money in my pocket at the end of it.

So, Go!

This working for yourself thing is hard. It’s so very important to be good at what you do, and yet being good at what you do has so little to do with being able to pull off working for yourself. On one level, I wish more people would do it — I believe it achieves a far more efficient distribution of skills and labor while allowing for some serious specialization. On the other hand, the instability of it ranges from mildly uncomfortable to downright terrifying, and so reality dictates that most people will wander back to the full-time world soon enough, and the independent thing will be but a blip on the resume. I, though, feel lucky I’ve made it this long.

On speaking at the 2009 jQuery Conference

2010-08-11T00:00:00Z

This post is really old! I've kept it around because it may still be interesting, but many things may be out of date.

One of my personal goals for this year was to start being part of the solution to the dearth of female speakers at tech events. Though I’ve talked at a couple of smallish local events over the past few months, this past weekend I got to do it in a big way: I presented a talk on using objects to organize your jQuery code to an audience of around 100 people, more by far than I’ve ever spoken to before.

[This post isn’t so much about the talk itself as my first experience with talking at a conference. If you’re interested in the talk, I encourage you to check out the slides, links, and code at the link above.]

I decided I wanted to try to talk at the jQuery conference after I saw the initial very smart, very male speaker lineup. I submitted my talk based on an article I wrote earlier this year, and by the time it was all said and done, mine was the second most popular topic and I was slated to have 30 minutes in “the big room.”

There is something sort of out-of-body about that moment when I am standing in front of a roomful of people right before I talk — I had it when I gave my first Refresh talk, when I taught my first jQuery class, when I spoke at my first BarCamp RDU, and yet again this weekend. For that moment, in my head, I am a complete and utter case, and can’t quite fathom that I thought this was a good idea. And then I start talking, and then it is OK. And then when it’s over, people clap, and I like that part.

Back when I set out to start speaking more, I decided to take an improv class. For six weeks, we practiced being spontaneously funny, and at the end, we got up on stage in front of a bunch of strangers and tried to do it for real. Knowing what that feels like — what it feels like to run up the aisle like you’re excited when really you’re terrified because you’ve never done this before and in real life you sit at a desk all day and talk to no one and what were you thinking? — makes the thought of talking to a bunch of strangers about what you actually know how to do seem like a completely reasonable thing.

My experience this weekend was nothing short of excellent — people I barely knew rallied around me throughout the weekend to help me improve my presentation (most notably Chris Williams, organizer of JSConf, to whom I owe many thanks for all the images — especially the Liger). The audience graciously tolerated the part in the middle where I had to leave the podium to (very publicly) blow my nose. People asked great questions, and audience members gently pointed out things I might want to rethink. With the exception of one creepy off-the-wall comment about my “fine-boned features,” the reaction was overwhelmingly positive.

Reliable sources told me that of 300 attendees, approximately 282 were men. I was the only woman to submit a talk. So this is the part where I encourage other women to do the same. I think women, on the whole (of course there are exceptions), are way more inclined than men to think they aren’t good enough speakers, that they don’t know a topic well enough to tell it to other people. Two truths: one, the speaking skills of the speakers I’ve seen have been all over the map; two, you’d be surprised how much you actually know about a topic, especially given the right audience. Go speak at a small event — a local meetup, a Refresh, even a lunch-and-learn at your office. Get to know the people who do speak at events, and discover that they’re people just like you. Go out on a limb and try something that’s

On rolling your own

2010-08-11T00:00:00Z

This post is really old! I've kept it around because it may still be interesting, but many things may be out of date.

There’s been a lot of activity around my last post, On jQuery & Large Applications. A number of people have asked me why, exactly, I’m so opposed to using jQuery as part of a roll-your-own solution.

To answer that, let’s start by looking at (some of) the pieces my ideal large application solution might include:

DOM manipulation tools
Ajax tools
A dependency management and build system
Clear patterns for code organization, such as namespaced modules
An inheritance system, preferably one that offers multiple inheritance, for sharing code across modules and staying DRY
A non-DOM-centric, loosely coupled API for communication between modules
A widget system that makes use of the inheritance system, with lifecycle management (setup/teardown) and templating
A system for maintaining templates separate from JavaScript while interning them into the build to eliminate HTTP requests
A system for abstracting RESTful server communication
For a UI-intensive project, a rich widget system pluggable to arbitrary data sources and designed with an easily-extended API
For an enterprise project, a11y and i18n provisions, as well as clear patterns for file organization

To all of you who have said you can do this yourself, you win. I can’t argue with you without sounding like I’m saying you’re too dumb, and you’re probably not.

But here’s my question: why? What, exactly — exactly — do you gain by putting all of this together for yourself, rather than using the pieces you need of a toolkit that provides all of this out of the box? Because here are a few things I think you lose:

Integration. Is your abstracted RESTful data API designed to talk to your widget system that’s designed to talk to your template system? That’s hott. Now what about when a new version of one of those components comes out that violates an assumption you made?
Maintenance. I heart the good folks who have put together individual answers to these individual questions, but they have no obligation to continue being the good folks they are, and they certainly have no obligation to do it on any sort of schedule. Remember all those plugins that broke with jQuery 1.4? That sure was fun.
Documentation. I’m going to grant you, right now, that jQuery is one of the best-documented JavaScript libraries out there, hands down. But what about all these other pieces you’re putting together? Especially the ones you really are rolling on your own, like that templated widget thing that communicates so nicely with your abstract data API. There are a wealth of resources for understanding, troubleshooting, and using these pieces in established toolkits. Where will the next developer turn when they have questions about yours?
Experience. Like I said, I get that you’re smart. Possibly smarter than me. That’s cool. But are you smarter than the combined wisdom of a team of people that has been thinking about these questions for years? Are you sure your solution has thought through all the questions they have?

I’ve noticed that, in the conversations I’ve had the last few days, it seems to fall to me to “prove” that a roll-your-own solution that includes jQuery isn’t advisable. Perhaps that’s fair — “you started it!”, you might say, and that I did. But simultaneously, others argue that jQuery never set out to answer these questions, and so it’s not jQuery’s fault that people are trying to use it in ways it wasn’t intended to be used. I have waited in vain to hear a compelling reason why jQuery should be part of a large application solution, to hear why I should recommend a roll-your-own solution that includes jQuery to my clients. The extent of the argument seems to be “because I like it, and it doesn’t force me to think a certain way.”

No one puts baby in a corner. Got it. But the straw man-ness of this argument has me, literally, chuckling right now. Let’s not confuse a mythical one-size-fits-all solution with a toolkit that provides, well, tools. Tools to do all sorts of things, tools meant to work together, tools developed and tested and maintained by a whole big team of smart people, tools that are, actively, being used in really frigging big, really frigging enterprisey applications.

I very purposefully didn’t propose a particular alternate solution in the original post, but it’s hardly a secret that my personal favorite, of late, has been Dojo. Not because it purports to solve every problem or prescribes how to solve them, but because it gives me so many tools to use to solve a given problem. Time and again I find that “Dojo already did that” — they already wrote the tool I’m wishing I had. Now I don’t have to write it, and, perhaps more importantly, I know it was written to work with all of the pieces I’m already using, and when I use it I’m not risking duplication of code or a lack of testing, maintenance or support. Win.

But let’s be very clear: no one’s forcing me to use that component! No one is forcing me to do things a certain way, any more than jQuery is “forcing” me to think of my application entirely in terms of the DOM. I can write my own component if I want, or use someone else’s if I want, or change it a bit if I want! For example, on a current project I pulled in mustache.js because the project had a lot of templates that had already been written to use it. The brilliant thing, though, was that integrating mustache.js into dijit._Templated instead of the standard templating system was trivial. That component, and all the others in Dojo and Dijit, are architected explicitly not to be one-size-fits-all. They provide a rock-solid base for large application development, for getting up and running quickly using a bevy of ready-made solutions, but also provide so many extension points that you can turn those solutions on their head if you want or need.

Garann Means, whose blog you should be reading, took a bit of issue with my original post in model-view-controller and comfy clothes.

I do agree that it benefits everyone to be working in the same setup and making use of tools that have been vetted by geniuses whose entire job is to create such things. But I’m dubious about any approach which comes too close to promising one size fits all. If you’ve ever sewn a dress, you understand that one size fits all is technically possible, but some people are going be left with a lot of excess while others will scarcely be able to breathe.

Carrying on that metaphor, these pieces provided by Dojo — or any other comprehensive toolkit, for that matter — are but starter patterns, and thread, and scissors, and pins, and a sewing machine, and OK I’m stretching the metaphor, now, but my point is they’re definitely not finished one-size-fits-nobody garments. On the other hand, if I decide to use jQuery in a large application, it can feel like I’ve been given a black marker and some of that crinkly brown paper, and now it’s up to me to draw a pattern and then come up with all those other pieces, too. Intellectually interesting and pleasingly crafty, perhaps, but not particularly efficient, sustainable, repeatable, or maintainable.

So again I ask, in all seriousness and in hopes of fostering a good discussion: Why? jQuery provides you with DOM, Ajax, and event management tools, but little else. There are tools designed for building large applications, designed to provide all of the pieces I want and so many more it’s not even funny, and they provide you with DOM, Ajax, and event management tools, too. What’s the compelling case for rolling your own solution that includes jQuery instead?

On jQuery & Large Applications

2010-08-09T00:00:00Z

This post is really old! I've kept it around because it may still be interesting, but many things may be out of date.

Update: I’ve written a separate post on the wisdom of rolling your own large application toolkit that incorporates jQuery.

I’ve been thinking a lot lately about JavaScript applications. As my skills have evolved, I’ve had the privilege of working on more actual applications, and I’ve gotten further and further from clients who want to add a bit of Ajax or bling to an otherwise fairly traditional web site.

The most interesting applications I work on are client-side intensive: the server is responsible for providing data as JSON to the client, and most everything else — templating, state management, data management, site navigation, and of course user interaction — is left to the client side.

It’s a lovely way of writing an application. There’s no need for me to touch server-side code; in some cases I work with a server-side developer to decide what the data they send will look like, but in others I just take what an API already provides and make it work. I get to use the same templating framework across projects, regardless of server-side technology, and I can prototype complex interactions before the server side even exists.

This is a land where HTML, CSS, and JavaScript are almost all you need, and I like it. I’ve become a firm believer in moving giant hunks of functionality that used to belong to the server over to the client. For a variety of reasons, I think it’s clear that this is where most interesting web development is headed, to the extent it’s not already there.

This style of building an application changes the front-end development game. In fact, “development” may no longer be an adequate description; we’re moving into the realm of engineering, here. We’re not using JavaScript to add a bit of bling to our sites — a slideshow here, some Ajax there — we’re architecting an application, damnit. We can’t just write some procedural code that binds a bunch of anonymous functions to some events and call it a day; if we do, I can tell you from experience that we’re going to end up with a steaming pile of unmaintainable crap.

Among a host of questions presented by these sorts of applications, some of the most interesting to me are:

What are the units of functionality that will make up the application?
How will those pieces be organized into units of code?
How will those pieces communicate with each other?
How will dependencies between components be expressed and managed while adhering to the principle of loose coupling?
How will components manifest themselves in the DOM? Do they need to?
How will we persist data across URL and page loads?
How will we manage communication with the server?
How will we make sure users only see the data they’re allowed to see?

At the risk of making a broad generalization, this isn’t the way today’s average JavaScripter learned to think. The mantra of jQuery, the most popular JavaScript library on the internets, is “get some elements, do something with them” — perfectly terrible preparation for analyzing an application from a perspective other than the DOM. And, IMHO, therein lies a tremendous problem.

As more and more application logic moves to the browser, I’m eager to see the JavaScript community rise to the challenge, but instead it feels like the opposite is happening. People with little understanding or appreciation of these questions are taking on projects that demand these questions be answered. The result is a land of fragile code that gets the job done while giving the finger to the next developer; a land of code so tightly coupled, so deeply beholden to the DOM, so blatantly not reusable or extensible or maintainable as to render every subsequent commit a complete crapshoot, as liable to cripple the application as not. The viability of the project is threatened, and so is the reputation of JavaScript.

We are better than this. JavaScript, even that old-fashioned browser kind, is a language worthy of respect, not a thing to be dreaded. But — and here’s the sentence I have struggled 10 months to realize and an hour to write: in order to prove that we are better than this, we must make abundantly clear to the budding developers, to the project managers, to the enterprises, to anyone intending to build a remotely complex JavaScript application, that there’s more to JavaScript than jQuery. The questions are bigger, the answers more complex, and the relevant skills, alas, a bit harder to come by.

We have to make clear that, in fact, jQuery is but a hammer. When it comes to building these intensively client-side applications, we’re talking about building skyscrapers, for god’s sake. The problems solved by a hammer are the least of our concerns.

It was just a few months ago that I gave a presentation on building large jQuery applications. I emphasized jQuery’s role as strictly a DOM and Ajax tool, and demonstrated a few other tools — John Resig’s simple inheritance, James Burke’s RequireJS dependency management and build tool, Jan Lenhardt’s mustache.js — that one would want to bring to the table for such an undertaking.

But to what end do we assemble said hodgepodge of tools? Is it just so we can continue to “use jQuery”?

jQuery’s API is, indeed, dead-simple, but we are smart people! We are building skyscrapers! When it’s time to write a complex application, and we need all of these things that jQuery doesn’t offer, can we not learn to use another hammer — learn that dojo.place('<div>I am new!</div>', oldDomElement, 'last') means the same thing as $('<div>I am new!</div>').appendTo(oldDomElement) — if learning it gives us access to legions more functionality than jQuery even aspires to provide?

Do we assemble this hodgepodge because finding jQuery developers is perceived as an easier task than finding practitioners of another library, even though someone saying they “know jQuery” is little indication that they will know how to work with the assembled solution?

Do we do it for the plugin ecosystem — full of code of varying quality and maintenance — even though many of the large application needs addressed by those plugins are addressed by other libraries as well, and sometimes better?

And when we do it, when we assemble this collection of tools ourselves, what risks are we accepting? What price will we pay down the road to maintain three or five or 10 different pieces from three or five or 10 different authors, with different release cycles, no guarantee of compatibility or maintenance, and no central project thoughtfully considering their future?

I’ve wrestled with these questions for months, agonizing during sleepless early-morning hours over how to advise clients on the answers. I’m the co-host of yayQuery, a contributor to the jQuery Cookbook, and, I’ll venture to say, a decently respected member of the jQuery community. I did not arrive at this conclusion lightly, and I have few illusions it will be well-received, or even heeded.

But I’ve grown weary of people championing a tool that simply does not answer the big questions I see in project after intensively client-side project. I’ve grown weary of those same people dismissing tools that answer those questions handily and have been answering them for a while now. I cringe when clients tell me they’ve chosen jQuery because it was “easy,” and then watch them predictably struggle with all of the questions it does not answer. And I’ve found I can’t continue to bite my tongue when people recommend jQuery as an enterprise-grade solution while failing to acknowledge these questions, let alone answer them*.

I do not want to see jQuery go away. The simplicity of its API was undeniably instrumental in the rise of JavaScript as a language these last few years. It is a perfect gateway drug, and I greatly enjoy watching people transition from “get some elements, do something with them” to the elegant patterns of JavaScript itself.

jQuery is an entirely appropriate answer to so many questions, but it falls so short for large applications, forcing you to assemble such a tenuous toolkit of your own, that it simply isn’t a viable answer — or, in my opinion, part of an answer — for large applications. If we hope to continue to gain respect as a community, we ought to admire jQuery’s immense contributions, but we must not be afraid to accept and make very clear its limitations. We do otherwise at our peril.

*An aside: To its credit, JupiterIT has put forward JavaScriptMVC, the only substantive attempt I’ve seen at answering these large application questions using jQuery. I applaud them, but fear their efforts will continue to be somewhat isolated without the support and endorsement of the wider jQuery community. If you have read this far and still have your heart set on a jQuery-centric large application solution, you should by all means take a look at JavaScriptMVC.

Rebecca Murphey

Eng ladders, promotions & glue work

all too well.

Writing effectively in software engineering organizations

Important purposes, essential questions #

Writing for the record #

Templates, patterns, and prior art #

A high-leverage activity that gets easier with practice #

Raspberry Pi setup notes

Set up for headless use #

Connect & update & install an editor #

Create a new user #

Oh My ZSH #

Python 3 #

Node #

Maybe #

Couldn’t I just save a copy of the SD card before I screw it up again? #

On micro-frontends

What the platform does #

Technical challenges #

Non-technical challenges #

To micro-frontend or not? #

Surprises and opportunities #

In summary #

Effective storytelling for internal platform teams

Know your audience #

Communicate value, not effort #

Get creative in how you communicate #

Know your lines #

Everyone has a part to play #

Wow, that’s a lot. #

Notes on setting up a JS project, circa 2016: ESLint

Closing Thoughts #

Notes on setting up a JS project, circa 2016: Babel and ES6

Closing Thoughts #

Notes on setting up a JS project, circa 2016: Webpack

Setting up Webpack #

Setting up the Webpack dev server #

Adding an npm script #

Closing Thoughts #

Indeed

Speaker economics

Chrome HTTP/2 Log Parser

Building for HTTP/2

Deploying JavaScript Applications (2013) #

Splitting it up #

Versioning #

Push: Because you downloaded scout.js, you might also like ... #

The elephants in the room #

For Consideration: HTTP/2-friendly deployments with HTTP/1.1 support #

Reality check #

Further Reading #

Thanks #

Five questions

Question 1: What are we doing and why? #

Question 2: How could it go wrong? #

Question 3: How will we know if it’s going wrong? #

Question 4: What will we do if it goes wrong? #

Question 5: Is there an “undo” button? #

A Baseline for Front-End [JS] Developers, 2015

JavaScript #

Modules & Build Tools #

Testing #

Process Automation #

Code Quality #

Git #

Client-Side Templating #

Node #

The End #

Writing Unit Tests for Existing JavaScript

Step 0 #

Mocking: Squire.js #

Spying: Sinon #

What to test ... and not #

Spreading the love #

Further Reading #

Show & tell at the Times Open Science Fair

A Baseline for Front-End Developers

JavaScript #

Git (and a Github account) #