90

A friend of mine is working in a company which is developing and maintaining a set of mission critical products. If one of those products fail at 3 AM, it is mandatory to do something immediately to get it back online, including solving the issue in the source code itself.

The company is now searching for a developer capable of solving issues related to the development itself. The friend will be in charge of the technical part of the interview, and asked me for hints.

Actually, a good candidate must be able to solve technical problems under severe stress. If the application stops working on the server of a customer, the company is paying a huge penalty per hour of downtime, meaning that the developer will know that while he is currently working on the issue, the company loses thousands of dollars. There may also be indirect pressure from angry customers¹.

How can I emulate or test such a stressful situation during an interview?

All the usual elements used during the interview to test how an interviewee would handle the stress² seem too small compared to what the person would do once hired, i.e. being required to rush in the middle of the night to the workplace in order to solve a bug as fast as possible.

Asking for previous similar experience seems harsh too. I'm not sure whether it is frequent to find workers who had worked in such frightening conditions.

So how can I gauge this skill during an interview? How can I know someone has this ability to work under stress before they're hired?


Update: more than two years passed since I asked this question. I kept regular contact with my friend, and was glad to hear a lot of good news. The company had a chance to hire some talented persons who took a bunch of good ideas and pushed them up to the management which was clever enough to accept them, by trusting those people unconditionally.

Among the changes, the development workflow was entirely redesigned, with the introduction of DevOps which works surprisingly well. Other measures related to code reviews, tests and the quality of code ensured that the product became progressively better. Now the developers are not afraid any longer to push their changes to production, because they are pretty sure those changes won't break anything.

There were difficult choices to do as well. Several programmers and managers were fired, because of their lack of skills and their unwillingness to learn, including a manager who were in this company for twenty years. Also were painful the decision to outsource a few jobs.

This being said, the company became much more mature, and, more importantly, much more pleasant place to work. Nobody screams at developers because of a regression they introduced, because there are none (at least not severe enough), and nobody should wake up at 3 AM because a problem was discovered and should be solved ASAP.

Surprisingly, the solution was very simple: the company now have three teams in three countries around the world, geographically chosen so that an evening in one country corresponds more or less to the morning in another country (there is, I believe, a three hours gap at some point that they have to manage by asking employees to start to work later than usual and to end later as well). This allows the permanent availability of skilled personnel whenever a problem occurs.

The only problem is that a bunch of services is still hosted in-house. This means that the company should have system administrators at reach day and night, on location, in case where a problem occurs to the data center itself (UPS failures, flood, servers starting to shut down unexpectedly, etc.)

This problem is about to become obsolete. The company is migrating its services to Amazon EC2, which appears to be slightly more expensive, but also much more reliable: a good price to pay.

I'm both happy and sad: happy to hear that the company is doing well, and sad telling myself that I focused my attention purely on the hiring process itself, instead of having a broader look at the problem, as any developer is expected to do. I had all the necessary info to think about a larger picture, but I haven't seen that the solution to hiring persons ready to wake up at 3 AM is to simply make the original problem irrelevant, would it be by outsourcing or by migrating to cloud computing.

This could be a good lesson—at least I consider it like one for me. Instead of looking for management-centered solutions to technical problems which often make employees unhappy, one should look at technical solutions to management problems. This case is a perfect illustration of that.


¹ Developers have a workspace separated from support department, but at night, I'm pretty sure that if an issue is found, there would be only two people in the company's building: the developer and the person from the support. While the support person will handle all the calls outside the developers workspace, the two will still talk together, so the developer would know that the customer is expecting a quick response.

² For example asking the interviewee to read or write code during a limited amount of time, or asking technical questions without telling if the answers are right or wrong, etc.

Arseni Mourzenko
  • 4,883
  • 4
  • 22
  • 31
  • 62
    :: reminded of the interview in Swordfish :: Whatever makes you think that putting someone under that much pressure gets things fixed quickly? Honestly, if they're willing to take a call at 3am and get it fixed at all, you should be thanking them for taking the job. – pdr Oct 04 '12 at 10:44
  • @pdr: This is exactly my opinion. As a side note, the job is well paid. And when I say "well", it's out of proportion compared to what I receive as a freelancer. – Arseni Mourzenko Oct 04 '12 at 10:50
  • 26
    Well, ok, so they are "thanked," which is good. But still, I am that guy who'll take a call at 3am and fix something, but if you keep calling me and telling me how much money you're losing because I haven't fixed it yet, I'll go back to bed. :) – pdr Oct 04 '12 at 10:57
  • 1
    @pdr: again, I completely agree with you. Personally, I wouldn't even take a job where someone can call me at 3 AM. – Arseni Mourzenko Oct 04 '12 at 11:00
  • 3
    @MainMa - Being on call does give one an incentive to get it right though. If you do your job well, you should never need to be called at 3am. In the last two years I've only had to deal with 6 call outs and most of those were dealt with over the phone or via remote desktop. – Mark Booth Oct 04 '12 at 11:25
  • @MarkBooth, I partially agree with you in regards to your last statement. There are cases, however, when the dev team highlights for problems that might occur down the road but still the stakeholders don't want to accept these suggestions (due to some business rules, budget, or whatever reason). In these specific cases, the dev team could've done a great job and the deliverable will fail constantly. Unfortunately. – Tiago Cardoso Oct 04 '12 at 13:04
  • 3
    I made some edits to clarify and highlight the question itself and removed a little backstory; I felt it was clear enough how significant the high-stress part of the job is. – Rarity Oct 04 '12 at 13:26
  • Added an answer that covers Oded's answer, but also adds in the effect of word usage when 'pitching' the opportunity to prospective employees. – Anthony Miller Oct 04 '12 at 16:27
  • 8
    I think this is more of a systemic problem with the way the company does business and not a hiring/interview problem. Before hiring anybody for the "hot-seat", this organization should be looking at ways to address both the symptoms and the root cause of these problems. What support and training are being provided to the people that do these jobs? After these triage events, is there an effort to debrief and perform a retrospective (lessons-learned exercise)? Or are hapless people just thrown at the problem and forced to be "accountable"? – Angelo Oct 04 '12 at 16:38
  • 72
    The person fixing the problem at 3AM isn't costing the company money -- the person who left the problem to blow up at 3AM is costing the company money. The person doing the fixing has an opportunity to reduce the cost to the company, but the fault for those costs lies elsewhere. – David Mackintosh Oct 04 '12 at 17:22
  • 4
    Also I think if I was being interviewed for such a position (having worked them before) I would first ask. I assume I will have a senior member of staff managing the customer while I work on the problem? – Preet Sangha Oct 04 '12 at 19:25
  • 8
    In the words of Gordon England, former Director of Avionics at General Dynamics / Fort Worth Division: "What would you do if he was in the hospital?" – John R. Strohm Oct 04 '12 at 19:46
  • two words: cage match – Ken Liu Oct 04 '12 at 19:48
  • 5
    Try making the candidate code while crawling under a barbed wire as machine gun bullets zip past above his head and a drill sergeant screams profanities nearby and mortar rounds going up around him. Well, you get the picture. – Hyangelo Oct 04 '12 at 19:58
  • 2
    Quick question: why do you think that "it is [not] frequent to find workers who have worked in such frightening conditions?" I mean, they're not debugging a Patriot missile battery while enemy Scuds are incoming. The worst case scenario is that they screw up badly and get fired. Most people have worked in jobs where they could conceivably screw up badly and get fired. And, come to think of it, if the job is so crazy demanding that it's hard to find people qualified to do it... well, that makes it rather less likely that they'll get fired, doesn't it? Have you really thought this one through? – Evan Harper Oct 04 '12 at 22:47
  • 2
    @MainMa You should add that detail about the money in the question. Knowing that it's a high-paid position for your market alters my sense of what they're looking for, which before looked like somebody to take the heat for a total lack of accountability on management's part in the first place. – Erik Reppen Oct 05 '12 at 00:29
  • @MainMa: think outside the box people - ex-military. – Greg McNulty Oct 05 '12 at 22:31
  • 6
    @MainMa I'm ex-military, went through Desert Storm, went back as a contractor for Iraqi Freedom. What you think is extreme stress is just the tip of the iceberg. Try debugging after you've been knocked off your chair and have to handle minor cuts from broken glass. You're asking all the wrong questions, and you have a toxic environment to quality. Good decisions are made by not feeling the stress of the moment, but they come at a high cost. Why don't you have a product that alleviates the issue (downtime, bugs, whatever) instead of paying for them in flesh? – Edwin Buck Oct 06 '12 at 02:35
  • It sounds as if (a) your developers should be taking turns as on-call level three support, and (b) you may need more than one person on call at all times, rather than looking for a single miracle worker whom you expect to come in cold and dig you out of the hole you've created. If someone with that kind of skill and willingness to operate under fire exists, you probably can't afford them. – keshlam Apr 05 '15 at 17:59
  • 2
    @keshlam: fortunately, that's what the concerned company ended up doing. Surprisingly, the solution was very simple: they now have three teams in three countries around the world, geographically chosen so that an evening in one country will correspond to the morning in another country. They have still to have system administrators on location to handle problems with servers and the data center itself, but they are migrating to Amazon EC2, so soon, this won't be a problem any longer. – Arseni Mourzenko Apr 05 '15 at 19:41
  • I know this recommendation is coming in late, but couldn't you just, y'know, ask the interviewee how they'd handle the situation, and in general how well they handle stress? – Zibbobz Apr 06 '15 at 14:40
  • 1
    How do you stop your finger from hurting when it gets hit by a hammer is to stop hitting your finger with a hammer. –  Apr 06 '15 at 23:02
  • 1
    Thanks for the wonderful follow-up. This is all good advice for any company to follow. – Bill Leeper Feb 20 '17 at 20:37
  • 1
    One way to deal with the original situation is to ensure, organizationally, that the person who gets up at 3am to fix the problem is not the same person who might have created the problem. In practice, it means not having the original developers also responsible for the 3am support side of things. This only works in bigger outfits, but it takes out the inherent stress with the support person also feeling responsible for creating the problem in the first place: extremely stressful, and liable to lead to big mistakes...I've been there on this, and it was awful/scary! – Pete855217 Oct 06 '17 at 04:07
  • 1
    How rigorously is your code being tested? Post-compilation code doesn't just work and then suddenly fail, the flaw was already in there at compile time. Requiring developers to jump up at 3AM (so much so that it's a cornerstone of their duties) seems more like it's covering for an incomplete testing suite than something that actually requires a developer (as opposed to IT support). – Flater May 25 '20 at 16:57
  • 2
    Thanks for coming back and sharing the real solution post maturity and growth! – mxyzplk May 06 '22 at 12:55
  • 2
    +1 Updates are great, and this is the best one I've ever seen. – Daniel R. Collins May 06 '22 at 19:20

18 Answers18

72

How to emulate, at best, a such stressful situation during an interview?

You don't.

That's what probation periods are for - to see how the person handles the pressure for real and if they don't then they do not pass their probation.

So long as it is clear in the interview process what the work requires and that there is a probation period the result of depends on performance under stress, the person being interviewed will know what they are getting themselves into and the company insures itself against hiring someone unsuitable long-term.

Oded
  • 22,035
  • 5
  • 87
  • 99
  • 13
    +1 Probation periods or contract to hire are the only way to test if a person can handle the stress of being on the job – maple_shaft Oct 04 '12 at 11:29
  • 2
    @maple_shaft - I would argue that prior success in a similarly demanding position would be a fairly strong indicator as well. Probably with less risk to business too... – Telastyn Oct 04 '12 at 13:32
  • 1
    @Telastyn - But how do you prove prior success in the interview room? – Oded Oct 04 '12 at 13:34
  • @oded - How do you prove prior success with any job? If they were there for a long time then they probably did an adequate enough job. – Telastyn Oct 04 '12 at 13:38
  • 2
    @Telastyn - Not always an indicator. Sometimes the opposite - see the dead sea effect. A probation period is the best insurance against those that talk the talk but can't walk the walk. – Oded Oct 04 '12 at 13:46
  • 1
    @oded - Sure, sticking around can be the opposite. The interview should focus on that previous experience to see if it was a good experience or prolonged stagnation. For the sort of situation where an outage is high business impact (and thus high stress) it seems to me to be too risky to depend on probationary periods (and their added stress) to evaluate candidates. – Telastyn Oct 04 '12 at 14:05
  • 1
    You might substitute "probation" with "training". This kind of problem is faced by network operators (for example long haul telecom). Such networks are maintained 24-7 by staff who are expected to respond to problems and are measured in terms of seconds. Customers of these networks have service level agreements that put the company on the hook for big money in the event of unscheduled down-time. The way they hire operator personnel is by training candidates for weeks and then evaluating them prior to putting them on a live network. – Angelo Oct 04 '12 at 16:53
  • @Angelo - true, but the question is about assessing someone for existing skills, not about training them up to have those skills (in this case the skill is working well under pressure). – Oded Oct 04 '12 at 18:52
  • If you are going to hire me with a "probation period" then you'd better be prepared to pay me a contractor's pay rate, since "contractor" is the word I hear when I hear the phrases "Contract-to-hire" and "Probation Period". So many employers use these phrases and then offer the same pay they would a full employee hire. I once told a potential employer that I would be fine with a contract to hire position but the pay rate was going to be 50% higher and their reply was, we don't want you to get used to the higher pay so we can't do that. Needless to say, I didn't take the job. – Dunk Oct 04 '12 at 22:05
  • 10
    Emergency on-call expectations are one thing but if I got the sense a company was more focused on the employee's high-stress 3am aptitude than his ability to avoid such issues in the first place, there'd be a me-shaped hole in your wall in short order. Don't ever forget that unless you're scraping at the bottom of the barrel, developers have options. – Erik Reppen Oct 04 '12 at 23:33
  • except that probation periods are exactly not the time to put someone in that high stress on-call pattern as to do that job properly demands in-depth knowledge of the system they're going to be supporting, knowledge they won't have yet. @Dunk most jobs I've had such on-call duty was part of the job description, as is a probation period of several months (this is in fact so normal here there are government limits on the duration of it), usually on-call time is compensated on a lower rate for inactive hours, overtime pay while actively handling a support call. – jwenting Oct 05 '12 at 06:21
  • 1
    @jwenting - I don't think I've had a job yet where I haven't been thrown in the deep end during my probation period, usually due to resource pressure elsewhere, but my current job is there first (in almost 20 years working for UK companies) where I have actually been explicitly compensated for out of hours support calls. – Mark Booth Oct 05 '12 at 09:40
  • 1
    @MarkBooth maybe I got lucky, maybe corporate culture here is different, maybe it's because when I've had such work it was as a contractor rather than an employee. And there's a difference between being thrown in the deep end and being left dangling there. A proper process would have you run a period of support work under supervision of a more experienced person for example, acting as a mentor and source of wisdom (or at least knowledge). – jwenting Oct 05 '12 at 09:54
  • @Dunk I agreen only partially: it's perfectly normal and fine for any job to have a small probation period, but what you said is perfectly fine for companies who try exploit this and have long "probation" periods. – o0'. Apr 02 '14 at 11:38
  • I disagree. If you need probation period to evaluate a candidate skills, your interview process is deeply flawed. Probation period are for rare cases, and not a replacement of good interview process. –  Jun 22 '18 at 20:44
55

Screening

You can't screen for candidates who would react well under the specific stresses of your environment. At best you can select candidates who have demonstrated capability during high stress positions in the past.

Even then however, as in finance, past performance is no guarantee of future performance.

Transparency

The most important thing is to make it plain during the interview that a candidate will be put in this position. You need to ask how they feel about it and let them ask any questions they might have.

If they are not prepared to work under these conditions, it is better to find out in the interview than later, when their job performance fails to meet expectations, they have to take time off work to cope and/or resign to find a less stressful job.

† I have worked somewhere this happened to a colleague.

How different people react

Different people react very differently to different kinds of stress. While many people would get very stressed in the situation you describe, other might be decidedly relaxed about their employers or customers losing money.

In fact, it could easily be the case that while the stressed developer might work faster, they might also make more mistakes and thus the relaxed developer might work more efficiently overall.

Role-playing as an interview technique

If you really want to find out how someone might react, you could role play out a typical situation, but this would be considered by many a rather extreme form of interviewing for a software development position (though not for some jobs like sales or call centre staff).

It can never be fully representative of the situation they are likely to find themselves in however - after all, many people will already be stressed by being in an interview situation.


Though your situation may feel extreme, it is actually fairly common.

Many of us work in environments where a fault in our software could cause financial loss or even injury, and being expected to be available on call to deal with incidents is a common requirement for a software developer in many sectors

Riot
  • 113
  • 5
Mark Booth
  • 5,199
  • 1
  • 35
  • 54
  • 20
    Cannot agree more with a "the relaxed developer might work more efficiently overall." part. To be successful on such position is to have a lot of confidence. Confidence is what differentiates one developer who is calmly fixing the problem line.by.line. from another who would spend half of the time stressing out and jumping up and down trying to get himself together. Confidence may come from having heaps of experience “been there done that, seen that many times before and still alive”, from a seniority, (continued) – PeterT Oct 04 '12 at 13:54
  • 2
    PeterT's comment continued: "when one knows that he can always “shut the xxxx off and do this because I need it now” almost to anyone who is getting on the way of the urgent fix but mostly if one knows that he is employable and should something go really wrong (otherwise the company would have such position open) and he is to take the blame, and face the “consequences”, he would be able to find new job fairly easily." – Rarity Oct 04 '12 at 15:38
  • 4
    Agree with the "Though your situation may feel extreme, it is actually fairly common". I have hardly ever worked in a programming job that didn't carry the stress of having to diagnose and fix a problem on a live system that was costing people money. – Carson63000 Oct 05 '12 at 02:57
  • 1
    @Carson63000 or putting lives at risk... working in a medical environment can certainly add stress to decisions! But I agree completely – Jon Story Feb 19 '15 at 10:21
44

What some people forget is that an interview works both ways. While you are interviewing a person for a role, they are interviewing you to see if you are worth working for.

Your simulation aside, if you described as above my first question in the interview would be "Why do you only have one developer for a mission critical system?", also "Why don't you have fail-over / clustering set up?".

I would also ask how often would you expect me to be woken up at 3AM? Do I go to work the next day if I spent 3AM-6AM working on an issue? Or am I required to be on call 24/7? What happens if I become sick and something happens?

So I would seriously question if such a company would remain viable if it had a single point of failure which I would be interviewing to be.

Any serious professional knows money is being lost in downtime, and explaining the stipulations of the contract they will work to that.

Simon O'Doherty
  • 6,049
  • 2
  • 24
  • 41
  • 10
    well said. Any such system will need at least 3 people at each position in the team. That way one can be sick or on vacation and there's still 2 people to take turns running the graveyard shift. – jwenting Oct 05 '12 at 06:25
  • "I would also ask how often would you expect me to be woken up at 3AM? Do I go to work the next day if I spent 3AM-6AM working on an issue? Or am I required to be on call 24/7? What happens if I become sick and something happens?" these are very important questions, I'm glad someone pointed them out. – o0'. Apr 02 '14 at 12:15
14

Don't tell the employee it's a stressful environment.

As others have stated, create a probationary period, setup the environment that the employee will be in, and watch his/her performance. If you tell them

You're going to be in a very stressful work environment.

That automatically gets the prospective employee to cringe at the opportunity, and it just sounds like he/she will have a horrible time working for your company. No one wants to be put into that kind of situation.

You can inform them that

Our work is challenging and demanding.

So instead of giving the message that 'your life is going to suck here', you're giving the message that 'this role is very important', and will give the prospective employee a sense of pride for the position.

EDIT: P.S. NEVER TELL AN EMPLOYEE THAT THEY ARE MAKING THE COMPANY LOSE MONEY! It's not constructive and does nothing for either you or the employee. That includes making any program/diagram showing the negative effects of their 'lack of urgency'. Find ways to motivate and not scare the employee to creating positive performance.

EDIT2: When you tell the employee "Our work is challenging and demanding", watch their reaction. Some will slump in their seat, draw their shoulders in, and their facial expression may change. This shows a lack of confidence.

If they show a sense of seriousness or excitement when you say this to them, their confidence levels are at least there and it puts them into the mental position that they need to step up to the plate if they want to work here.

Anthony Miller
  • 947
  • 1
  • 5
  • 12
  • 6
    this is absolutely the best answer! – Alex Gordon Oct 04 '12 at 16:24
  • 16
    It kind of sounds like you are saying they should lie to prospective employees. Also, if someone told me "You're going to be in a very stressful work environment" I would want to hear why it was stressful. If someone told me "Our work is challenging and demanding" I would wonder exactly how they were trying to snow me. – psr Oct 04 '12 at 18:27
  • There's a difference between "Lieing" and putting a situation in a different "Perspective". Lieing is saying "it's like any other programming job". Whether the job is "Stressful" or "Challenging" is purely a perception of that job. By human nature, if you tell someone that what they are about to do will cause undue hardship on them, the obvious reaction is to abort. If you tell someone that what they are about to do will strengthen them and test their resolve, the reaction is to prepare and strengthen oneself for action. – Anthony Miller Oct 04 '12 at 18:49
  • 4
    Be careful not to misrepresent the situation. "Challenging and demanding is rather vague" and could mean any number of things. – Ken Liu Oct 04 '12 at 19:52
  • So can "Stressful". Again it's about perception of what the work is. However, calling something stressful is negative while calling something challenging and demanding is positive. – Anthony Miller Oct 04 '12 at 20:31
  • 1
    @Mechaflash: Then focus on specifics; In this case challenging and demanding means 24/7 availability, and ability to respond to outages within a short time frame. Don't hire a guy who's going to quit/be let go within 3 months because you've misrepresented the job. That's a brilliant way of wasting hiring costs and generating ill will. Probation is cheaper, but not free. – deworde Oct 04 '12 at 23:08
  • Challenging is good. But who wants demanding? – Erik Reppen Oct 04 '12 at 23:37
  • 3
    I'm conflicted about this answer. On the one hand +1 for Don't tell an employee that they are making the company lose money but -1 for Don't tell the employee it's a stressful environment. I can think of few things more demotivating than starting a job only to find that it was different to your expectations of it. This will just lead to unnecessary churn. – Mark Booth Oct 05 '12 at 09:44
  • Think psychology. Whoever said misrepresenting the position? It's for an initial statement. You always tell them what the job requirements are, i.e. 24/7 availability, the company's promise to its customer and what your role plays in that, etc. It's the introductory to all of that is what the statement is for. Telling them that it's 'Stressful' starts them in the mindset that they're not going to like it, thus they'll be closed off to the rest of the info. Stating it as 'Challenging and demanding' gives them a sense of pride for the position because it sounds important. So they'll be open. – Anthony Miller Oct 05 '12 at 13:10
  • 12
    I hear "challenging and demanding" the way I hear "fast-paced environment." Both mean "our project management sucks, and we expect heroics from you to compensate." – Amy Blankenship Oct 10 '12 at 02:19
  • Downvoted because describing it as a "challenging and demanding environment" without providing specific details (like being on-call 24/7) is really misleading. You clarified in your comment that you'd also tell them the job requirements, but that's missing from the answer itself. Not every interviewer does clearly lay out the responsibility and expectations of a position, so I don't think it can be assumed. – Kelly Tessena Keck Apr 03 '15 at 19:53
12

If it were me doing the hiring, I would look first for candidates who had done night time support in the past. At least they know what they are getting into and whether they would be willing to do it again.

I would also look more favorably on people who realy really know their stuff. If you have in depth knowledge, fixing things on the fly at 3 am is much less stressful than for the person who doesn't know where to start. I would look for people who go the extra mile in their current or past postions. People who know thier data model inside out, people who are organizaed and can put their fingers on teh tools and techniques they need to solve a problem quickly and people who have a track record of being a successful troubleshooter.

In addition to some programming problems, give them some troubleshooting tasks to perform. Take one of those things that went wrong at 5am and see if they can find the problem. More important than finding and solving the problem (they are afterall not familar with the code base) is how they go about looking for the problem.

Techniques I learned to test how they handle stress in the interview include (use these with care, they may turn off some good people): Use a panel of people all of whom do not react favorably or unfavorably to the answers given. If you give the person no clue what you are thinking, the interview becomes much more stressful. Ask some questions that you don't expect people to get correct. (it's a plus if they actually do get those correct as well) Disagree with a correct answer.

HLGEM
  • 142,324
  • 26
  • 258
  • 516
  • 1
    This is a good answer. I would start by having them prove they can solve problems or similar problems that were already fixed ( in a short amount of time ) during perhaps a probation period. Depending on some factors you could do this during an interview also. This would prove if they are able to problem solve in a short amount of time. – Donald Oct 09 '12 at 17:13
9

1 - Narrow it down

"Able to handle stress" is a very vague concept. A person who gets stressed with very fixed work hours and excessive management supervision, may find a job where they have to react immediately, think fast, and communicate well with minimum supervision to be an ideal job.

Figure out what traits you really need and what the context of the work will be. I'd suspect you want someone who can:

  • be a jack of all trades, and able to dig into anything, strong debugging skills across complex systems

  • readily ask for help and give good feedback/guidance/status since in a crisis communication like this is key

  • work largely unsupervised with minimal oversight

  • take ownership of the problem and take the crisis seriously

  • accomodate the demands of critical outages with their work/life balance

  • can perform sustainably - what I mean is, a one-hit wonder who can solve the crisis fast, but who is such a jerk in the process that no one will pick up the phone the next time he calls isn't going to do you any favors in the long run.

That's going to vary depending somewhat on your conditions. For example:

  • can they fix the crisis from home or do they need to drive into worK? If they work from home, your communication needs go up, but your "can get to the office in X minutes or less" requirement is not necessary.

  • will a manager be called in, too? If so, maybe you don't need someone who can communicate to ALL stakeholders, but the relationship between the person and his manager is critical, since they have to be a very strong team.

None of this is what I'd call "stress" - it's all a question of whether this person can perform well in the environmnent being provided.

2 - Get demonstration of skills, observe for characteristics

You can witness some of these skills in simulation - for example, giving an ambiguous problem which only sketchy details and focusing the candidate on time pressure is going to tell you something. But more important is how do they solve the problem. Be aware of the little things, when you observe, that show that the candidate is optimizing for the crisis, not just proving that he is smart.

3 - Be clear, have a discussion

When it comes to the question "is this guy going to be able to sustain these demands or will it collide with personal life?" - there's a lot of questins you can't ask in some countries like - "are you married?", "got kids?" and "are you pregnant?"

Instead, it's best to ask candidates if they can meet the demands of the job and ask them to give you supporting cases:

Question: We need this person to be on site to fix bugs within 15 minutes of being notified. Can you meet this, if so, tell me how?

Maybe they live close by, maybe they are willing to sleep in their car when they are on call - it only matters that they can meet your requirements in a way that you believe.

4 - Have a trial period

There's a strong case to be made that even if your company doesn't have a policy of probationary periods, you haven't hired for this type of job before, and it's different and could use a promotion period.

bethlakshmi
  • 80,080
  • 5
  • 163
  • 308
7

The military train people not to panic. It's all about training and experience; you don't hire somebody who doesn't panic, you train them. (Or somebody else trained them.)

Repetition builds confidence, and so you need at least one highly experienced support person. Nobody has confidence before they get experience.

McGarnagle
  • 726
  • 2
  • 6
  • 15
peterretief
  • 169
  • 6
  • 3
    Would it not be easier to get people known to handle stress well, or filter for those qualities rather than train them? Not everyone has the budgets and requirements of armies. Nor do all people who start such training end it or come out of it stress resistant. – Oded Oct 04 '12 at 14:54
  • @Oded: Most companies have solved this problem in saner ways (by eliminating the 3AM-human-intervention-needed failures) so you're not going to find many people with a proven trackrecord. Armies of course have less flexibility in picking work environments so they do have to adapt. – MSalters Oct 05 '12 at 07:55
  • 3
    +1 - Having been in the military, I can say that this is indeed true. A company that does mock emergency preparedness drills will be much more prepared for when all hell breaks loose. – jmort253 Oct 07 '12 at 21:23
  • @Oded if there was a test to identify people known to handle stress well or filter for those qualities, then the military could in theory screen for those people and then reduce or eliminate boot camp. – emory Oct 07 '12 at 23:47
  • @emory - That's my point. There is no such test except for putting people under real stress and see how they cope. That's what basic training is all about. – Oded Oct 08 '12 at 08:55
  • The military trains people to accept the command of the commanding officer and do automatically act upon their training. This is achieved by drills. Lets be honest there are tons of ways to prevent the stressfull situation the author is asking about, since its not actually life and death, perhaps those solutions should be implemented first. – Donald Oct 09 '12 at 17:20
7

Unless you plan on interviewing them by calling them at 3 AM on a random morning and grilling them on tough interview questions (which I am not advocating), I'm not sure you can figure this out in an interview.

If they have prior experience in this type of environment, that is certainly a plus but isn't a true indicator. Survival in that type of scenario is primarily based on their in-depth knowledge of the system being supported. If they really understand the system they have a chance of succeeding, if not, don't put them on-call in the first place.

I work for a large finance company with global offices and a global customer base. Sometimes I'm the guy who gets that 3:00 AM call. Once I wake up, my success is tied directly to my comfort level with the sub-system which is failing. If it's something I've worked on, I'm in pretty good shape. If it's an interface with another subsystem, things get harder.

The best you can do to handle these stressful situations is to make sure you team (or at least the ones who will be on call) are capable people who truly understand your application. That comes from experience with your system and you can't get that out of an interview.

cdkMoose
  • 17,840
  • 5
  • 49
  • 73
  • correct, and you can't figure it out during a probation period either. During that period the new hire is learning to use and maintain the system, learning the company culture, he doesn't yet have the knowledge needed to perform emergency maintenance on it (unless maybe that maintenance equates to telling someone to reboot the server, but that's not something that other person should have to be told to do, it should have been written down in HIS checklist of things to do in case of problems). – jwenting Oct 05 '12 at 06:29
6

May I answer your question with another question?

Are surgeons tested to work under stress before being hired? Do they jump out of bed an run through the streets when they are on call and there's an emergency? Don't they get quickly used to stressful situations even when every minute counts and take them with complete calm?

The only way to get things done is to pretend it's not an emergency and focus on what you need to do.

What you want is an experienced and intelligent person that can solve the problems that are expected to happen. I would focus on testing that and just making clear to the interviewed what his tasks and related conditions are.

yannis
  • 5,314
  • 7
  • 41
  • 70
Pablo
  • 131
  • 2
  • Junior doctors train under these conditions. As such it seems to me that that is a what probation is for. – Preet Sangha Oct 04 '12 at 19:22
  • @PreetSangha - Even Doctors during their internship period know when they are all call. They are not expected to solve problems. When a doctor does their internships, they have council, and are not normally the final decision makers. It also depends on several factors, most doctors are not placed in the ER, unless thats what they want to do. – Donald Oct 09 '12 at 17:16
5

Hire people who are ex-military with combat hours.

I know you are in France, but there are plenty of soldiers there who have had countless hours being shot at. Your company losing money is probably not going to ever be as stressful as the situations they have worked in before. It may be a small percentage of combat soldiers who program, but they are out there.

national emblem of France

Mithical
  • 225
  • 1
  • 4
  • 10
Greg McNulty
  • 3,330
  • 22
  • 26
  • 6
    +1 - Nothing I ever do will ever be harder than being in the military. To ex-soldiers, having to stay up all night to fix something is nothing compared to someone who had to stay up for days digging holes and doing nothing but train, train, train in life and death situations. – jmort253 Oct 07 '12 at 21:26
  • @jmort253: hey wehre did my plus 1 go!?...yes, so true, not sure how this is not the accepted answer but I guess people don't really get it unless they have worked with or as military. – Greg McNulty Oct 11 '12 at 21:56
4

Planning a work environment for stress I believe is the wrong approach to start with. This is not a war, it's IT. There is no reason not to do serious planning to reduce stress by having sensible contingencies in the event of an emergency. Also constant practice in the form of drills also help greatly reduce the possibly of it being a stressful situation. If there is fear, for example, that a server might go down and cause a loss of revenue, then have back-up servers and mirroring the operations.

Practice taking them in and out of service, have mated pairs and off-site help in stand-by in the case of a real emergency like an extended power outage or fire. If you are going to plan to have a stressful environment and look to hire people who can handle that, then from my experience you are going to find people who don't plan, get bored very easily and do nothing until there is an emergency so they can play "fireman" and be a hero. This is a bad business practice and environment for IT and I would be doing and have done, everything possible to avoid such a problem.

Rarity
  • 6,179
  • 6
  • 40
  • 60
Edward
  • 462
  • 2
  • 5
2

There are some good answers here. I would like to add that there are some specialised support roles ( I am thinking Trading Floor support in banking / hedge funds ) which are very time critical and high stress since not having a timely resolution would lead to a direct financial loss. It is not easy to emulate such a situation, but your friend could look for people with similar experiences.

I have worked in such roles and even now work very closely with support personnel who have similar jobs. The best support people tend to:

  • Learn from previous experiences: If a particular batch job fails and if for the first time it takes them three hours to figure it out - they document and communicate the solution so that the next time only takes minutes to resolve the situation.

  • Be familiar with basic support tools: They should know how to search for errors in a log file, find out the disk utilisation on a server, how to check which processes are using which port to communicate, etc. Knowledge of a scripting language like Perl or Python is usually a good sign for a support person.

  • Be able to communicate clearly: They should be able to explain the behaviour of the system clearly to the developer and to the business person. They will need to coordinate any emergency bug fixes, server restarts, etc., so being able to communicate clearly is very important

Your friend could check for these attributes in potential candidates for the role.

Luhar
  • 772
  • 4
  • 6
1

I can't work under stress at all. I solve this problem by not letting anything stress me. If I had to fix a problem at 3am in the morning that costs the company $10,000 an hour, that's not stressful. It's inconvenient because it's 3am. The money isn't my money, so that doesn't stress me. If I had a supervisor wanting a status report every ten minutes I'd tell them to **** off ahem go and pester someone else, I have work to do.

I can work focused, and I know what my most effective speed is. It's not my fastest speed, because that is where mistakes are made. It's the speed that leads to fixing the problem quickest in my experience.

And if it's so expensive to have delays, you'd have two developers coming in. It will save you money.

gnasher729
  • 169,032
  • 78
  • 316
  • 508
1

Another thing to consider. As an IT professional, I've ended such interviews when I felt they were planning games with me. The management might think they are clever, but no good person is going to put up with this sort of nonsense. Serious advanced planning is the only way to run things properly.

Edward
  • 462
  • 2
  • 5
0

Since when have stopping a company losing some money be of that much importance? You have to find a better reason for the system to be fixed quickly and then interview for people that deeply care about that reason.

You are looking for someone that thinks the customer is doing a worthwhile task and will feels a sense of ownership of the problem.

Hopefully when the job is explained, most people that are not willing to do it will withdraw from the process. However some people will lose benefits, or have their family require them to take the first job they can get, or just be so fed up with their current job that they will take anything to get out.

So ask the person lots of questions over at least 2 interviews spread out over at least 2 weeks about how they will cope practically with it – you don’t care what the answers are, just that you are making them think about it.

Then only take on someone that loves their current job with a current job that does not pay much less. You are trying to select someone that has positively chosen to take your job.

Ian
  • 1,387
  • 9
  • 10
0

I think if you are fully upfront about the demands of the position, many candidates will weed themselves out. Some will have family situations that are not condicive to being on call.

Also, you need to rethink how you plan on reacting to these situations. Changing code is not the same as rebooting a server.

-3

Depending how much freedom you have with the interview process - you could possibly perform psychometric testing. @Oded I thought about a sedentary test which you could perform. Chess is possibly an answer, or speed chess. A timed game against a grandmaster is one way to put pressure player/interviewee, especially considering the interviewee will probably be under the impression they have to win.

There have been some interesting studies into chess, it's stress implications, and the various monitoring devices which you could use. Stressed Chess, and "The stress of chess players as a model to study the effects of psychological stimuli on physiological responses: an example of substrate oxidation and heart rate variability in man". These are only ideas, if indeed the need to monitor a persons ability to deal with stress is paramount.

Personally though, I think talking to person, finding out about their previous experience (work, and life) is probably more applicable. If the person is technical competent than they are less likely to suffer from stress, than someone who can deal with stress and is technically less competent. Testing for stress by chess, could be a little too bizarre?

wonea
  • 147
  • 4
  • 6
    What if the candidate never played chess before? I would say you would be testing for stress during chess play rather that stress while fixing urgent bugs... – Oded Oct 04 '12 at 14:04
  • If they can't play chess then it would not work, also they could easily get out of the test by saying they can't actually play chess. Also could you draw parallels between chess induced stress and bug fixing? Almost certainly not. – wonea Oct 04 '12 at 14:14
  • 2
    If they've never played chess before, and you gave them the impression that they needed to beat a grandmaster in a game of speed chess, I'd say you'd see some genuine stress in that interview situation. :-) – Carson63000 Oct 05 '12 at 02:58
  • Now that would be a great sketch! – wonea Oct 05 '12 at 08:18
  • I have a decent online rating and though not in some organization, reliably beaten people with decent FIDE ratings over the board. I would not be stressed out against a GM, because I know winning is highly unlikely. Also, while timed games are stressful, I wouldn't compare it to the situation described in the question at all. – bytepusher May 06 '22 at 10:39
-7

Why not simulate that environment exactly?

For example, let's say you have a number of problems for them to solve during the interview process. Before you get to these problems, show the candidate something like the following:

Dollars Made $137.03 | [New Problem] button | Potential Lost $0.00 | [Problem Fixed] button [10-Minute Penalty] button

Explain, as you did in your question, that the company makes thousands of dollars on a given project per hour. Thus, when there is a bug in the project which causes it to go down for an extended period of time, there is the potential for a great amount of profit loss. Furthermore, while some people may thrive in this type of environment, others would find it too stressful. Thus, in order to find out how they deal with this type of stressful situation, there is a portion of the interview process dedicated to replicating this environment to see how they will deal with it.

At this point, ask them if they are OK with this type of test. If not, then skip it and just ask the programming questions as normal and judge them on the rest of the interview, taking into account that they avoided the stressful situation. If so, click the "New Problem" button. Explain to them that a bug was found and the product is now down, and that in an actual situation, the clock would be running even while they are trying to figure out what the problem is. Also, explain that even though there is a time-crunch, fixing the bug properly is the bigger goal here. Then hand them the problem, preferably on a piece of paper with no further explanation and sit back, observe, and answer questions for them as you deem appropriate.

When they are finished with the problem, click the "Problem Fixed" button and evaluate their solution to make sure it meets the requirements. If it does not, click the "10-Minute Penalty" button, explaining again that fixing the bug properly is the bigger goal, and have them continue working as the Dollars Lost continues accruing.

Repeat for the remaining problems, asking them if they are ok to continue before starting the next problem, and possibly asking them how they are dealing with the stress of the situation. Be empathetic, understanding that this is not a fun process to go through (for most).

At the end you will get a concrete number for total dollars lost on all problems, as well as the qualitative experience of how they perform under pressure.

Here's the editable link for the simulation if you want to tweak it to better match your case.

Mithical
  • 225
  • 1
  • 4
  • 10
Briguy37
  • 225
  • 1
  • 8
  • 10
    And now let's pretend for a moment that the candidate is an adult professional who is good enough to have other options. Interviews are a two-way street and you're giving me a very strong message about the people I would be working with. I'm not sure it's the message you intend. – pdr Oct 04 '12 at 15:25
  • 1
    @pdr: As an adult professional, I want to know what to expect from a work environment for any job I consider. The above is a realistic simulation of that work environment. Thus, I would be in a better position to know whether that position would be a good fit for me after going through the above interview process. The right candidates will thrive under that stress and will accept the position if they deem the compensation appropriate, and the wrong ones will hopefully be weeded out either by their choice or the interviewer's. – Briguy37 Oct 04 '12 at 16:02
  • @pdr: Is it a waste of times to ask good programming questions? No! Why, then, is it a waste of time if you ask those same questions in this format? As I said in my answer, those who answer foolishly will be penalized, so it finds people who are both quick and accurate under pressure. Also, the simulated situation is losing money, the developer is minimizing that loss, so maybe there should be a goal time for each problem and a savings/loss if you finish before/after that time. Also, the interviewer's setup could be more thorough to explain the background of why before starting the test. – Briguy37 Oct 04 '12 at 16:27
  • 5
    Sorry, deleted that comment just as yours came in, it was not well phrased. My point is that you are sending me a message that you prioritise firefighting over not having open flames near gas. That is a bad message to send. Don't scare off good programmers and you'll end up with a quality product that rarely goes down, rather than a bunch of people who can fix it under pressure and have to, often. – pdr Oct 04 '12 at 16:30
  • 2
    Your approach is logical, but it is not practical. Yes, you want to find the right candidate. However, showing them how much they can screw up is an easy way to deter good people who do . The only people that you will attract are those who are OVERLY confident. That's another type of animal you don't want employed. – Anthony Miller Oct 04 '12 at 16:42
  • The position itself should attract employees, not the interview process. Thus, if there is a sufficiently high demand for the position (and that is the big if), including this as a "stress-test" portion of the interview process will not wean out many good candidates. For example, if NASA wanted to hire a top-level-on-call-guru to handle all "mission-critical" software problems, you can bet that most/all applicants who passed would stick around and probably even understand why it was given. However, if Mom-And-Pop Software did the same thing, the smart ones would run for the door. – Briguy37 Oct 04 '12 at 20:34
  • 5
    I make a lot of decisions about what the job is going to be like based on how they interview. People acting like you're lucky to be there and treating the interview like a one-way phenomenon is always a bad sign. I don't care if it's NASA, Google, or Pixar. – Erik Reppen Oct 05 '12 at 00:16
  • @EricReppen: Is the above really treating them like they are lucky to be there? That's one take on it, and obviously the down-voters' take, but it is far from the real purpose of determining whether or not they are up for the job's challenges as soon as possible. Honestly, this shows much more respect to a candidate than several other approaches here with more votes, as it is honest, forthright, and does not waste a candidate's time or hide important facts about what the job environment will be like. – Briguy37 Oct 05 '12 at 13:14
  • Thanks for all the comments. I see that the original test left/would leave a bad taste in many people's mouths, so I've updated my answer to give the candidate a better feel for the purpose of the test before starting, give them a chance to opt out of it, and to lessen/remove the sense that the company/interviewer is a killjoy. – Briguy37 Oct 05 '12 at 16:28