Job Search 2009

2009-11-30

I recently started a new full-time job writing software. Here is how that came about:

Thu Oct 1: Returned from trip. (I’d previously decided to look for work upon returning.)
Thu Oct 8: Sent Peter a blurb to go out with Friday’s SF Bay Area Caltech Monthly Luncheon invite: Grad seeks software engineering job to supplement income. 10+ years experience. (I mistakenly thought this might motivate me to update my resume before the luncheon.)
Thu Oct 15: Attended luncheon. Discovered that one of my fellow alumni and former co-workers is now a hiring manager with a possible opening. (Nobody else ever contacted me about my blurb.)
Mon Oct 19: Beat resume into submission. Emailed link to former co-worker and a few other close friends (less than 10, certainly). Linked to resume from Facebook status.
Tue Oct 20: Visited 5 friends in 3 Bay Area cities. Exchanged more email.
Wed Oct 21: Posted final draft of resume, incorporating various suggestions. Took first call from a prospective employer (specifically, from a recruiter in a Human Resources department).
Thu Oct 22: First technical phone screen. Thus far, I’d heard from 6 potential employers in total.
Wed Oct 28: Heard from 10th potential employer. (I’d elected to pursue 7 of those 10.)
Thu Oct 29: First on-site interview.
Thu Nov 12: 9th and last on-site interview. (3 of the 9 were second visits.)
Fri Nov 13: Reviewed notes; deliberated; chose. Notified all 7 contenders. Drove down to sign employment agreement.
Sat/Sun Nov 14/15: Wrote to those who had requested a more detailed explanation.
Mon Nov 16: First day at the new job. (I took Tuesday off so we could celebrate our monthaversary as planned.)

Further factoids

I was referred to all 7 companies by current employees who know me. 5 were former co-workers. 2 were hiring managers; the rest were engineers.
All interviews were for a full-time position as some variety of software engineer, programmer, MTS, code monkey, or other equivalent term.
At least 3 companies asked me to sign a non-disclosure agreement prior to my first on-site interview.
I would rather not identify all 7 companies by name or group too much information by company. Sorry.
All 7 companies did some kind of work involving distributed systems running on clusters of Internet-connected computers running Unix-like operating systems. [So did various leads I didn’t pursue, for that matter.]
2 of the 7 companies had between 6 and 50 employees; 1 had 51 to 500; 2 had 501 to 5,000; 2 had 5,001 to 50000.
Locations ranged from San Francisco to Mountain View. [I live in Oakland.]
5 companies did phone screens, 4 of which were divided into a call from someone in HR and a call from an engineer with technical questions; the 5th was from a hiring manager who did some of both.
6 of the 7 companies invited me for on-site interviews. I visited 3 of those companies twice, for a total of 9 on-site interviews. I did complete a technical phone screen with the 7th company, but I don’t know whether they would have invited me on-site or not.
Technical phone screen start times ranged from 9 a.m. to 3 p.m. or later; scheduled duration was between 30 and 60 minutes.
Scheduled on-site start times ranged from 10 a.m. to 1 p.m.; scheduled duration ranged from 2 to 5 hours.
5 companies scheduled on-site interviews overlapping with the noon hour. Of those, 3 took me out to lunch; the other 2 had no lunch break.
2 different companies invited me to talk to an engineer based in Europe. One called me at home; I dialed the other from a conference room during an on-site interview. Both calls were in the morning, at 9 or 10 a.m. Pacific time.
My new job is at Blekko. No, I’d never heard of them either. No, I can’t tell you much about it.
Yes, I am still working at Shiny Devices. No, this is not a secret. Yes, I’d be happy to tell you all about it.

I like to think that I’m at least somewhat aware of how lucky and how privileged I am, and I might editorialize about that or some other aspect of this experience another time. For now, I thought it might be interesting just to list some of the questions I was asked. As far as I know, nothing written here is a direct quote from anybody. Any lack of specificity means that my notes or recollections are vague, or that I’m being lazy, or that I think the specifics are none of your business.

Preliminary questions

Here are some of the non-technical initial questions the HR people asked me.

Could you send me your resume? [This only happened once.]
If employed, can you submit verification of your legal right to work in the U.S.?
Of the specific job openings listed on our web site, which ones are you primarily interested in applying for? (You’d think I’d learn my lesson after being asked this once, but no.)
Are you looking for a full-time position? [I’d initially made noises about being open to part-time work.]
Have you worked here before?
Have you or applied for a position at this company before?
I understand that you were referred by X; is that correct? [A: Yes.]
Who else at this company do you know? [I was asked this at least 2 times, and listed several other people in each case.]
Where are you in your interview process? [I tended to interpret this as: “How many companies have you contacted, how many have contacted you, when did you last hear from each, and what stage have you reached in each case? And for that matter, which companies are these, and (for the more obscure ones) how big are they and what do they do?” But nobody ever pressed me for anywhere near that level of detail, so I never ended up providing it.]
What is your timeline? [A favorite! I tended to interpret this as being equivalent to the previous question or to “When could you hypothetically start work?”]
What salary do you require? How much did your previous employers pay you? [Only 1 company asked this during the initial HR call, but see below.]

I spent most of my time on this type of call repeating and expanding on the contents of my resume. For example:

What programming languages are you the most comfortable using? [I was repeatedly told that they were asking me this specifically in order to select an interviewer for the technical phone screen.]
I see on your resume that you worked for Pagebites as a System Architect. Could you talk a bit more about what that actually entailed? [Mostly for recent and/or relevant experience, but not exclusively. Sometimes I’d have guesses afterwards about which word in that job description had caught their eye. Several times people told me they were just curious about some not-so-relevant project, such as Shiny Devices or my Ditch Day stack.]
How are things at Shiny Devices? What was your reason for leaving Pagebites? [Mostly about current/recent jobs, but a few questions about older jobs too.]
How many employees work(ed) for Shiny Devices? Pagebites? SafeWeb? iSpheres? [Nobody asked this about Microsoft or Symantec. Strange!]
Didn’t our employee X (the one who referred you to us), also work at company Y (listed on resume)? Were you both there at the same time? [A: Yes indeed we were.]
Have you ever worked with MySQL? How comfortable are you with C? C++? Perl? Writing kernel drivers? [Some language/tool/technology that they used, or that was somehow related to something they did, often in a way that wasn’t obvious to me at the time.]
Which of the following 20 or so options sounds most interesting to you? Choose at most 3 or 4. [One place did this with groups I might be interested in interviewing with; another did it with areas of computer science, again in order to decide who might interview me.]
What interests you? [Open-ended version of the above multiple-choice question. I can’t think of any company who didn’t ask me something very close to this at some point.]

Technical phone screens

4 out of 4 HR people mentioned my interviewer’s name when scheduling the technical phone screen. In 3 of the 4 cases, I elected to webstalk my interviewer beforehand. I was moderately entertained in 2 cases, regretted it in 0 cases, and decided it had been useful in 0 cases. In the 4th case, I got a first name only (or forgot the last name, perhaps), did no “research,” and ended up getting a call from someone with a different first name. Go figure.

One engineer did ask me what language I wanted to use for the coding question (2 companies used web-based shared whiteboards to ask coding questions during the phone screen), and several spent a few minutes up front asking me to discuss specific projects I’d listed on my resume. All four spent the majority of the call asking me technical questions. At least one launched into a somewhat involved technical question before I’d uttered more than a few words; I had to ask for some details to be repeated so I could write them down. In 1 case I had no time to ask any questions whatsoever; the other 3 interviewers gave me a few minutes at the very end of the call. Several people implied that they thought I would be invited for an on-site interview, but 0 promised this, and some didn’t mention the topic at all.

Telephone interviewers from 2 different companies asked superficially similar questions about what happens when you invoke wget or curl on a URL. I believe both used http://www.google.com/ as their example. One interviewer seemed to be more interested in network and system level stuff, such as the sequence of calls to routines in the BSD socket API, e.g. gethostbyname(), socket(), connect(), and a few high-level details of DNS, HTTP, and TCP. The follow-up question in that case was a seemingly open-ended question about making a large download go faster; the specific answer desired apparently involved fetching ranges using multiple HTTP connections. The other cast a wider net, reflected in the original question’s phrasing: I was to describe what happens upon entering the command into bash running in an xterm on a Linux machine, which (once I’d summarized my previous spiel and realized I wasn’t done) gave me an excuse to talk about USB HID, Unix domain sockets, ptys, fork and exec… we ran out of time before I got started about ld-linux.so.2, which was probably just as well.

A few more discussion questions I was asked at this stage:

What do malloc() and free() do? Describe how you might implement them. Can you think of any ways to improve on that implementation? Follow-up: Where does the memory used by malloc() and free() come from? List a few advantages of virtual memory. [And so on. Memorable near-quote: “No, actually, I think a more pressing issue is that your malloc() sucks, if you’ll excuse my French.”]
Thread 1 requires simultaneous access to resources A and B; thread 2 needs A and C; thread 3 needs A, B, and C. How might one mediate access to those resources? [A: Guard all 3 resources with 1 mutex.]
Describe how to find the 10 most frequently-used words in a large corpus of text. Follow-up: What if your histogram is too large to fit in RAM? Follow-up: List some of the worst deficiencies of that approach. [A: Well, it needs to make at least two passes over the data, which is kind of irksome… oh, and it’s not guaranteed to terminate… yeah, that last is probably the worst, actually.]

2 different companies used a shared web-based whiteboard [my term] to ask me a coding question over the phone. I was warned about this ahead of time in both cases, so as to ensure that I’d be near an Internet-connected computer. Each web-based whiteboard was developed in house; neither was entirely glitch-free, but both were more than adequate. The 2 questions were as follows:

Write a function that sorts an array of integers in place. Is your function correct? Are you sure? What’s it’s asymptotic efficiency? Really? What’s the asymptotic efficiency of an optimal solution? [And so on.]
Write a function that finds the longest non-decreasing subsequence of an array of integers, e.g. 1 10 2 8 3 5 6 4 7 12 11 becomes 1 2 3 5 6 7 12.

[I managed to flummox both interviewers by stumbling upon novel and unexpectedly inefficient initial solutions, thereby cutting off the usual avenues for incremental improvement.]

On-site interviews

I was given a start and end time for all 9 on-site interviews (as well as a location, for the 6 first-time visits at least). I was never given a more detailed schedule ahead of time, though several of the interviews started with a meeting with a hiring manager or someone in HR who gave me the day’s schedule. Each company divided the interview into uniform 30, 45, or 60-minute slots. The only non-uniformities I can recall were either for lunch or (in 2 cases) to wait for an opening in a busy executive’s schedule. There was one interviewer per slot in all but 2 or 3 cases with 2 or 3 simultaneous interviewers, one of whom invariably did most of the talking.

The on-site questions didn’t really differ much from the telephone questions. There were fewer of the easy preliminary questions, but still quite a lot of questions about things I’d done in the past or might be interested in doing in the future. In fact, I’d guess that I spent more time overall talking about past projects than answering technical questions, though the proportion varied wildly between companies. Many of my on-site interviewers, even if they were engineers, asked only non-technical questions. Some, especially higher-ups, spent the whole time selling me on the company. Many, if not most, started by describing the company and their particular product or role. These descriptions didn’t usually overlap as much as they seemed to fear. Some of my interviewers were in a different department or group than what I was interviewing for; these tended to pre-emptively justify their presence. Almost all sacrificed at least a few minutes of their precious time slots for me to ask further questions. I mostly stuck with variations on “What do you, personally, do here?” and “What’s your typical day like?” In a couple of cases I asked somebody to sketch a partial org chart. Once I asked somebody to show me some code. [They did.]

On-site non-technical questions

Where do you see yourself in 5 or 10 years? Follow-up: Do you aspire to become a manager? [Fairly common.]
What would you do with your life, given unlimited resources (such as money and employees)? [A memorable one-time variation.]
Describe a past situation where you responded to a customer request / encountered an external obstacle / disagreed with someone about how something should be done / were introduced to a large pre-existing codebase. [And maybe a few more I’m forgetting. Once each, I think.]
What are your hobbies outside of work? [A: “Well… uh… um… I’ve been learning how to do a pull-up!” I was only asked this once during the entire process. Some might consider this a dodgy question, legally speaking. More on that below.]

On-site technical discussions

Have you used or do you know anything about mysql/Agile/perl/hadoop/xen/Linux drivers/OUTER JOIN/Erlang? [I’m not making any of those up. Yes, I was asked about some of those during phone screens as well.]
Which databases have you worked with in the past? When would you use a sub-select vs an inner join? How does a WHERE clause differ from the HAVING clause of a GROUP BY? [This interviewer mentioned having been tasked with asking me database-related questions.]
Define: object, inheritance. List some types of inheritance. [A: private/protected/public/default — not my first guess.]
How would you choose between Java and C++? [I managed to turn this one into an excuse to discuss the design of the Shiny Devices firmware configuration and upgrade utility for the 999th time.]
How might one design a distributed, transactional database? [2 times! Took up most of the interview slot in each case. Lots of whiteboard diagrams.]
Name some of Python’s built-in types. Which of those types are immutable? [Python became my stock answer to “What language shall we use for the following question?”]
Tell me about the database schema at Pagebites. How many tables were there?
What’s the most technically challenging thing you’ve done? [I only recall being asked this 1 time.]
What technologies do you like, e.g. Rails? What’s your favorite web framework for Python? [Interviewer liked Rails.]
What’s a TLB? [3 or 4 times at least!]
What kind of software load might run especially slowly under a hypervisor? [Once.]
When and how do you hand off code to QA? What test frameworks have you used? [Interviewer worked in QA department.]
What’s the return type of a constructor in C++? [Asked during a coding question.]
How might you go about finding the 1 million most common bigrams [pairs of consecutive words] in a 100 million document corpus? [I mentioned that another company had asked me something similar during a phone screen, and wished aloud that I’d researched it afterward.]
I know this isn’t especially relevant, but I’m curious: What processor did you use for your embedded Linux prototype? Was that an ARM7TDMI? Any on-chip Flash? How much RAM? How’d you get it to boot? [A rapid-fire barrage of questions which I cannot possibly hope to replicate here.]

On-site coding questions

Some of these were prefaced by a question about what programming language I’d prefer to use. Some were not.

Interviewers at 2 different companies asked similar questions about analyzing a hypothetical web server access log. Both interviewers invited me to write code on a whiteboard in any language I liked. Each line in the log file contains a fixed number of space-separated fields, always in the same order, e.g. URL, browser name, client IP address, seconds required to serve the request. In one case, the log was compressed with gzip (which made it easier to guess at the preferred form of my answer, I felt), and I was to count unique client IP addresses. [A: zcat access.log.gz | cut -d’ ’ -f3 | sort -u | wc -l] In the other case, I was to generate a deduplicated list of IP addresses that made at least one request requiring at least 2 seconds to serve.
Describe how you would test whether a number is a power of 2 in C? Follow-up: What does the following C function do?
```
int something(int N)
{
 int i = 0;
 while(N)
 {
   N &= N-1;
   ++i;
 }
 return i;
}
```
Follow-up: How might you modify the above function to compute parity?
This next question was framed in terms of a book recommendation algorithm, but was phrased in terms of a bipartite graph. [I was amused to hear those words come out of my interviewer’s mouth before we’d even gotten to the question.] You’ve got a list of people and a list of books, and you know which people have read which books. Given a person P, write a function in the language of your choice that returns the list of all books B such that P has not read B and B has been read by at least one person Q such that Q has read every book P has read. [My interviewer asked the question in a less confusing but equally exact way, and may even have sketched out a diagram.] Start by making and stating some reasonable assumptions about how the information is stored and how your function can retrieve it. You may assume the existence of lower-level data retrieval functions.
Write a function in the language of your choice that checks to see whether a string is a syntactically valid IP address.
Write a C++ prototype for a function that accepts an HTML document as input (as a NUL-terminated const char *) and returns a deduplicated set of link URLs as output. [A: void findLinks(const char* htmlIn, std::set<std::string>& linksOut);]
List some inputs for a test plan for a function that checks email addresses for syntactic validity. Follow-up: Write the function, in Perl.
Verbally sketch out the design for a class representing an email message, in Perl. Follow-up: Write code for the constructor.
Here is a printout of a Unix man page for strcmp() [and strncmp()]. Here is a ballpoint pen. Write code for strcmp() in the bottom margin. Follow-up: Does your implementation work correctly for UTF-8 strings? [A: Yes.]
I forget the setup for the next question, but I think it involved some sort of unnecessary apology/excuse for asking me to reinvent the wheel. My interviewer wrote the following on a piece of paper (with a large blank space between the curly braces, and not much space between the class declaration and the following statements):
```
class String
{


};

String s1("abc");
String s2;
s2 = s1;
cout << s2;
```
Describe what class members String would need in order for the four lines at the bottom to work. [Phrasing was a bit misleading, perhaps intentionally, as operator << needs to be declared outside the class.] Follow-up: Write a prototype for the assignment operator. Follow-up: Describe some potential pitfalls in the implementation of the assignment operator. [A: self-assignment]
You have a SQL table called “employees” with three rows and three columns:
```
id  name   salary
=================
1   Alice       9
2   Bob        14
3   Carol      12
```
Write a SQL UPDATE statement that changes Alice’s salary to 15. [I used the id column in the WHERE clause rather than the name, eliciting no comment before we moved on to another question.]
Write Python code to flatten a nested list, e.g. [1, 5, [8, 12, [17]], 3, [27]] becomes [1, 5, 8, 12, 17, 3, 27]. Follow-up: Does your code work for all inputs? [I thought mine would, but it turns out Python limits the call stack to 1000 levels or so, preventing my recursive solution from flattening deeply nested lists.] Follow-up: Write one that does.
For this next question, my interviewer first wrote and described a big nasty Perl data structure similar to the following:
```
my %people = (
  123 /*UID*/ => ["Alice", ["project 1", "project 2", "project 3"], 1234567890 /* birthdate as Unix timestamp */],
  456 => ["Bob", [], 1234565432],
  /* ... */
);
```
Write Perl code to print name, number of projects, and UID as three space-separated fields, one hash entry per line, sorted by number of projects (most to fewest) and then by name (in case of a tie). Follow-up: convert those dates of birth to human-readable form. [We didn’t actually get to that.]
Write a non-trivial function in the language of your choice to convert a string into a floating point number [or flag an error]. Follow-up: How might you go about testing the function you just wrote?
Define: hash table. Describe how you might implement one, assuming string keys and values. Now draw a diagram of the data structure you just described. Next, assume the following C++ declarations:
```
int hash(char* c) { /* */ }

class hash_table {
  public:
    hash_table(...);
    void insert(char* key, char* value);
    char* find(char* key);

  private:

};
```
Fill in the private data members. Now write the constructor. Now write the insert method. [A: See photo below.]

Homework

One company contacted me after an on-site interview to schedule a time to email me a coding question. I was given 2.5 hours to complete the task, or as much of it as I could. The task involved implementing a data structure (in C, using only the standard library, or in Java, using the standard library minus java.util). The detailed instructions included a list of operations, arranged in order of priority, that the data structure should support. [I didn’t get to them all, or even close. My submission was written in C and contained 4471 bytes in total. It was quite a stressful experience, in part due to my own foolish overscheduling, and I had mixed feelings about it. I later heard that it had passed “with flying colors,” which was nice to hear.]

Taboo questions

In about 2 instances, I elected not to answer a specific question about some aspect of something I did for a previous employer that I felt was proprietary. In those cases, I said that I’d rather not go into so much detail about X, offered an extremely vague answer, and tried to suggest a related area where I felt that I could go into more detail. So that’s been pretty rare for me. It may help that many of the projects I’ve worked on no longer exist, although I’ve been known to refuse to comment even in that case.

In California (and in the USA generally), certain interview questions are considered illegal. I got a few questions that brushed on these topics. A recruiter asked about my work authorization; I stumbled a bit and admitted that I was a U.S. citizen. Over lunch, somebody asked me whether I’d grown up in California; I mentioned Ottawa. As I understand it, employers are forbidden from using a few specific types of information to make discriminatory hiring decisions, even though I’m still technically allowed to provide such information. Even so, I wish I’d answered those questions in the spirit in which they were asked (“Yes, I am indeed prepared provide evidence of my U.S. work authorization if and when you employ me” and “Gosh, I don’t know if that’s really such a good topic for us to chat about in the context of a job interview here in these United States”), rather than accidentally putting a prospective employer in a situation that might cause unnecessary worry about legal issues.

Just as in every other job search I’ve done, all 7 companies asked at some point about my compensation requirements, my salary history, or both. In fact, at least 3 of the companies I interviewed with had a formal job application form (paper or electronic) including both questions, to be completed before receiving an offer, with some redundant but intimidating boilerplate at the bottom about how failure to answer any question on the form is grounds for rejection and/or future dismissal (usually right next to the “at-will” bit about how either party will be able to terminate the employment agreement without notice or justification). Totally normal stuff. However, as it happens, I can’t recall ever having disclosed my salary history or specific requirements to any potential employer, during this or any other job search. As far as I am aware, this has never been a significant problem for anyone, though folks do often seem to push back quite hard, for perfectly understandable reasons. While some companies played it cool this time around, I feel like at least 3 of the 7 acted sort of surprised about it, which was too bad.

Some of the same companies (and more besides) seemed to react somewhat negatively when I accepted Blekko’s offer without soliciting a formal offer from everyone else (or, rather, from the negatively-reacting company in particular), which also seems like kind of a shame. I suppose I could have gone to the other extreme and tried to spark some sort of bidding war. Imagine the flak I could have drawn by blogging about that!