Your advice, ethical dilemma - job interview.

Non-investing personal finance issues including insurance, credit, real estate, taxes, employment and legal issues such as trusts and wills
Post Reply
HurdyGurdy
Posts: 1172
Joined: Wed May 09, 2012 10:21 pm

Your advice, ethical dilemma - job interview.

Post by HurdyGurdy » Fri May 23, 2014 4:42 pm

Bogleheads, may I ask for your advice on this?

I may be having a job interview this coming week, for a data analysis job. For the job interview, I should present an analysis of a dataset they sent me, coded answers to a survey. They say: "Please know that the committee is most interested in the process you use to analyze data AND how you present it to others. We are less interested in the actual results from this survey". However, since I needed to understand the data, I asked them what the codes meant, and what was the survey about.

After seeing the original survey form, I realize that there are serious problems with the survey design, such that the answers were biased towards giving a positive evaluation of something that this office does. (I can describe the details of why this survey is biased).

I do need the job, but my instinct says that I should cancel this appointment. Data collection and analysis should be done with the highest ethical and technical standards. (However, it could be that the survey was designed by someone who was not my would-be supervisor.)

If I go to the interview, I feel I'd need to talk about the bias in the design of the survey.

Should I cancel the interview?
As a non-traditional and older student, I recently got an MS in Statistics. Are situations like these, common in the job market?
Should I just shut up, present as if all was good, and, if job was offered, do it for a while?
LadyGeek, should I change my login name?

HornedToad
Posts: 891
Joined: Wed May 21, 2008 12:36 am

Re: Your advice, ethical dilemma - job interview.

Post by HornedToad » Fri May 23, 2014 4:56 pm

Why would you cancel the interview? It sounds like you are someone who would be a good fit and they need you. Doing good data analysis and coming up with non-biased surveys is hard work.

What you should do is go to the interview and then:
1. Point out the bias imbedded in the survey.
2. Explain what causes the bias and the extent that the bias might affect answers (5%, 10%, undeterminable amount)
3. Explain how the survey could have been better worded to get more accurate answers
4. Then answer the rest of what they asked for by analyzing the biased dataset and explaining what the results would mean etc. just like if the survey was not biased. For extra credit, analyze a range of possible results if the survey was not biased.

This demonstrates that not only can you analyze data, you can interpret the data and survey design and present to the audience in a clear rational fashion how to improve future results without offending anyone. (Definitely take the without offending anyone approach in your presentation)

ResearchMed
Posts: 7579
Joined: Fri Dec 26, 2008 11:25 pm

Re: Your advice, ethical dilemma - job interview.

Post by ResearchMed » Fri May 23, 2014 5:13 pm

HornedToad wrote:Why would you cancel the interview? It sounds like you are someone who would be a good fit and they need you. Doing good data analysis and coming up with non-biased surveys is hard work.

What you should do is go to the interview and then:
1. Point out the bias imbedded in the survey.
2. Explain what causes the bias and the extent that the bias might affect answers (5%, 10%, undeterminable amount)
3. Explain how the survey could have been better worded to get more accurate answers
4. Then answer the rest of what they asked for by analyzing the biased dataset and explaining what the results would mean etc. just like if the survey was not biased. For extra credit, analyze a range of possible results if the survey was not biased.

This demonstrates that not only can you analyze data, you can interpret the data and survey design and present to the audience in a clear rational fashion how to improve future results without offending anyone. (Definitely take the without offending anyone approach in your presentation)
^^ This.

And ESPECIALLY the "extra credit" part. Make sure you label this as "hypothetical", as one can't know what the results "would have been" from this exercise.
But by pointing out possible differences in interpretation, based upon how the survey was conducted (wording, selection of respondents, response rate - and how respondents might vary from those who declined... or those who were never included in the first place..., etc.), you can show that *you* can "add value".

However, this assumes that YOU would have some input in the development of future surveys.
If not, you will regret being required to continue to analyze "bad survey data".
(And that also runs the risk that someone else - supervisor or other - might decide to rewrite the "results" without mentioning the caveats, etc. I know. In a legal case where I created and analyzed a data extraction "survey", from case records, the client INSISTED upon changing "there was no evidence in X% of the written case records that <a certain good action> had been taken" to "in X% of the cases, <a certain good action> was NOT PROVIDED". Two VERY different situations. I refused to "adjust" the wording. Legal client found someone else to "write" the final report. It was incredibly difficult for both me and the consulting firm. I felt like there had been very little point in all the care we had taken in constructing the entire survey instrument and in training those who reviewed all the records.)

RM

User avatar
Grandpaboys
Posts: 878
Joined: Sun Mar 04, 2007 9:16 am
Location: Texas

Re: Your advice, ethical dilemma - job interview.

Post by Grandpaboys » Fri May 23, 2014 5:25 pm

Looks like you are the man for the Job, good luck.
Good Day | GP

User avatar
rob
Posts: 2991
Joined: Mon Feb 19, 2007 6:49 pm
Location: Here

Re: Your advice, ethical dilemma - job interview.

Post by rob » Fri May 23, 2014 5:31 pm

HurdyGurdy wrote:I do need the job, but my instinct says that I should cancel this appointment. Data collection and analysis should be done with the highest ethical and technical standards.
Agree with above but your gota temper this part..... You've not been at this much hey, otherwise I would expect far more pragmatism...... I've been in this data world for a very long time and worked in a number of very large companies..... lets say quality is NOT job #1.
| Rob | Its a dangerous business going out your front door. - J.R.R.Tolkien

Gattamelata
Posts: 191
Joined: Mon May 05, 2014 7:41 pm

Re: Your advice, ethical dilemma - job interview.

Post by Gattamelata » Fri May 23, 2014 5:41 pm

HornedToad wrote: What you should do is go to the interview and then:
1. Point out the bias imbedded in the survey.
2. Explain what causes the bias and the extent that the bias might affect answers (5%, 10%, undeterminable amount)
3. Explain how the survey could have been better worded to get more accurate answers
4. Then answer the rest of what they asked for by analyzing the biased dataset and explaining what the results would mean etc. just like if the survey was not biased. For extra credit, analyze a range of possible results if the survey was not biased.
I like this suggestion a lot. My only suggestions is that you consider doing #4 first. They asked you an explicit question, and answering that should be your first priority. Doing otherwise suggests that you think you know better than they do how the interview should go. It might mean preparing two assessments: one that ignores the bias in the survey, and one that includes the bias in the assessment. In fact, that could be a grand slam interview presentation: "Here's my assessment of the data as presented. But after getting more information on the survey, I found significant bias. This is my assessment with the bias factored in. Here are the differences, and this shows the impact of the bias. Here's how I'd redesign the survey to remove the bias and increase accuracy." That way the arc of your interview starts with meeting their expectations, then moves to surprising them, then exceeding their expectations, and finally giving them something actionable. I think it might be hard not to hire that!

DTSC
Posts: 1180
Joined: Mon Oct 20, 2008 9:47 am
Location: Illinois

Re: Your advice, ethical dilemma - job interview.

Post by DTSC » Fri May 23, 2014 5:46 pm

Maybe they are testing your ethics

HornedToad
Posts: 891
Joined: Wed May 21, 2008 12:36 am

Re: Your advice, ethical dilemma - job interview.

Post by HornedToad » Fri May 23, 2014 5:47 pm

Gattamelata wrote:
HornedToad wrote: What you should do is go to the interview and then:
1. Point out the bias imbedded in the survey.
2. Explain what causes the bias and the extent that the bias might affect answers (5%, 10%, undeterminable amount)
3. Explain how the survey could have been better worded to get more accurate answers
4. Then answer the rest of what they asked for by analyzing the biased dataset and explaining what the results would mean etc. just like if the survey was not biased. For extra credit, analyze a range of possible results if the survey was not biased.
I like this suggestion a lot. My only suggestions is that you consider doing #4 first. They asked you an explicit question, and answering that should be your first priority. Doing otherwise suggests that you think you know better than they do how the interview should go. It might mean preparing two assessments: one that ignores the bias in the survey, and one that includes the bias in the assessment. In fact, that could be a grand slam interview presentation: "Here's my assessment of the data as presented. But after getting more information on the survey, I found significant bias. This is my assessment with the bias factored in. Here are the differences, and this shows the impact of the bias. Here's how I'd redesign the survey to remove the bias and increase accuracy." That way the arc of your interview starts with meeting their expectations, then moves to surprising them, then exceeding their expectations, and finally giving them something actionable. I think it might be hard not to hire that!
I agree with this. This is an improvement on my suggestion and would work out better. It's definitely a way to show significant value without upsetting them too much at the start like my proposal had

User avatar
WallyBird
Posts: 186
Joined: Fri Jan 10, 2014 12:17 pm

Re: Your advice, ethical dilemma - job interview.

Post by WallyBird » Fri May 23, 2014 5:52 pm

Unless you're very certain that what they're doing is illegal or highly unethical, I'd suggest you go to the interview. Present what they asked for, and include your observations about the data. There are diplomatic ways to do this. ("Based on my initial review of the data....)

It might be possible that they threw you some faked or altered data just for the interview.

You can also express your concerns at the end of the interview, when they ask if you have any questions for them. Or you can ask at a later stage. If it's really sensitive, try to talk 1 on 1 with the highest ranking interviewer. If the answer is unsatisfactory, you can thank them for their time and say goodbye.

If the data are genuine and the issue is real, then they should be impressed and grateful that you've brought it to their attention. If not, well, congratulate yourself for missing out on a job with Bernard L Madoff Securities.

If nothing else, figure you're getting a unique opportunity to test your interviewing skills.
"Look, sir, we can't just do nothing." | "Why not? It's usually best."

User avatar
Ged
Posts: 3632
Joined: Mon May 13, 2013 1:48 pm
Location: Roke

Re: Your advice, ethical dilemma - job interview.

Post by Ged » Fri May 23, 2014 5:54 pm

I would think that the office would want an honest view of what they do, not a biased one. If they don't, well maybe it's a place you don't really want to work for.

livesoft
Posts: 62855
Joined: Thu Mar 01, 2007 8:00 pm

Re: Your advice, ethical dilemma - job interview.

Post by livesoft » Fri May 23, 2014 5:59 pm

Where else are you gonna get a real practice interview?

If you cancel the interview, you will not get the job. If you go and be yourself, you don't have to accept a job if it is offered. An interview is for the prospective employer to figure out if they want to hire the prospective employee. It is also for the prospective employee to figure out if they want to work for the prospective employer.
Wiki This signature message sponsored by sscritic: Learn to fish.

Professor Emeritus
Posts: 2628
Joined: Mon Aug 13, 2012 6:43 am

Re: Your advice, ethical dilemma - job interview.

Post by Professor Emeritus » Fri May 23, 2014 6:08 pm

Treat it as a purely hypothetical data set, created for the purpose of the interview Discuss it in detail. then raise the validation issue i.e. what questions you would ask if it was a real data set.

cherijoh
Posts: 4989
Joined: Tue Feb 20, 2007 4:49 pm
Location: Charlotte NC

Re: Your advice, ethical dilemma - job interview.

Post by cherijoh » Fri May 23, 2014 6:28 pm

DTSC wrote:Maybe they are testing your ethics
I think it more likely that someone who didn't know what they were doing designed the survey. The free web-based free survey tools make it very easy to design a professional "looking" survey - so many companies no longer see the need to outsource their survey design to professionals.

bonaire27
Posts: 94
Joined: Sun Apr 27, 2014 12:46 pm

Re: Your advice, ethical dilemma - job interview.

Post by bonaire27 » Fri May 23, 2014 6:28 pm

If you had to ask to see the survey, then clearly this is not the focus of your interview. The reality is that once you leave those stat books, real world statistics are quite different. I suppose you can say that about any discipline. The academic world and the business world are very different animals. Whereas, if you were to take an academic position in stats, you could design and conduct analyses with very little noise. WHen you are doing stats in the business world the noise is everywhere from the higher ups with strong opinions, the funding is never enough, the data is dirty and many assumptions will need to be made. My point is that I would not consider this an ethical question at this time. You have no idea what went into this study design. It is likely a compromise of many forces. I doubt if there was some nefarious attempts in designing a bias questionnaire. If so, I doubt if they would be presenting it to you to analyze.

I am a fellow statistician. When I interview statisticians and when the other departments also interview the statistician the biggest question isn't whether you know how to check normality assumptions. I assume you can pick up a book and figure that out. We want to know if you can present statistics to a non-statistical audience. After all, that is what you will have to do. Let's face it, us statisticians can be a bit "nerdy", at least that is what the other departments think. So show them that you are comfortable with them and that you can take a complex analyses and explain it to non-statisticians.

You will find that you will never find the pure statistics you studied in your books. That is just a reality once you leave a "controlled" environment. You will have to compromise from time-to-time. You will need to show you can work in a team. I am not at all suggesting you compromise good ethical statistics, but I am saying that you will need to bend a bit to meet other's objectives as well. You will think you are in the right but guess what, they will think they are in the right as well. This is what you will need to learn to navigate.

So I would suggest you go to the interview and do the interview as they have asked. If there is time, maybe mention that it would have been "interesting" to ask a question a certain way as you are curious what that result would have looked like. If they ask why, then gently tell them that perhaps the question was a bit bias. But be careful. These guys very likely tried there best to do the questionnaire right. They probably spent considerable time putting it together as a team. Don't just walk in there and tear it apart. That isn't showing you can work well in a team. When you get the job and start doing good work, they will start asking more and more of your input. But first, get the job! :sharebeer

ABQ4804
Posts: 389
Joined: Sun Jul 24, 2011 4:08 pm

Re: Your advice, ethical dilemma - job interview.

Post by ABQ4804 » Fri May 23, 2014 6:35 pm

I agree with Gattamelata's approach, described above.
Last edited by ABQ4804 on Sat May 05, 2018 12:32 am, edited 1 time in total.

pkcrafter
Posts: 13126
Joined: Sun Mar 04, 2007 12:19 pm
Location: CA
Contact:

Re: Your advice, ethical dilemma - job interview.

Post by pkcrafter » Fri May 23, 2014 6:44 pm

I agree with the others. They may well know the survey was constructed with a bias, and maybe they want to know if you can find it--or they know it and want it that way. I must say that many surveys I've taken in recent years were clearly designed to promote a positive result. If that's what they are doing, then you can make a decision that considers the ethical question.

I suspect that a lot of the bias is built in because there is incentive for improving customer satisfaction, and what better easier way to do that than tweak the surveys for higher satisfaction.

good luck,

Paul
Last edited by pkcrafter on Fri May 23, 2014 6:49 pm, edited 1 time in total.
When times are good, investors tend to forget about risk and focus on opportunity. When times are bad, investors tend to forget about opportunity and focus on risk.

User avatar
LadyGeek
Site Admin
Posts: 49146
Joined: Sat Dec 20, 2008 5:34 pm
Location: Philadelphia
Contact:

Re: Your advice, ethical dilemma - job interview.

Post by LadyGeek » Fri May 23, 2014 6:45 pm

This thread is now in the Personal Finance (Not Investing) forum (employment dilemma).

Remember that the purpose of anything a company does; whether it's an advertisement, informational article (a.k.a. "white paper"), or survey; is designed to promote the company. Take the interview. Be constructive and show them how to make it both unbiased and more biased.

If you really think you won't be doing this job in the long term, then don't take the interview. The last thing you want to do is protest the content - you'll be out of a job in no time.
HurdyGurdy wrote:LadyGeek, should I change my login name?
If you asked, you are concerned. For peace of mind, PM me with a new choice. If it doesn't show up in Find a member, you should be good to go.
Wiki To some, the glass is half full. To others, the glass is half empty. To an engineer, it's twice the size it needs to be.

User avatar
LowER
Posts: 678
Joined: Sat Sep 29, 2012 12:43 pm

Re: Your advice, ethical dilemma - job interview.

Post by LowER » Fri May 23, 2014 7:21 pm

You can't get a job that you don't interview for when one of the requirements is an interview.

In my field, a common piece of advice is to interview with the least promising prospects first to get the jitters out of the way and get a feel for what folks are asking.

Unequivocally, go to the interview and learn about them, and more importantly, yourself. The more horrible the interview, the more valuable it can be to you, and the more calm and cool you will be in future interviews that go well.

I agree with the others, and to proceed in a non-judgmental, very non-judgmental way, which is much different than not expressing your opinion as you can express strong opinion in a non-judgmental way, which is an art and a skill - maybe this is what they are probing.

I think that your extra effort and honesty will be greatly appreciated, and don't forget some tactful humor, preferably minimally self deprecating (or profession deprecating) and salient, i.e.; a good joke about statisticians; there must be a few good ones, right? Maybe data ANALyst,.... or not.

Just go be you! If that works, awesome. If not, it should have never been, and break ups on the first date are much less painful than after a few years. You're an older grad, use that.

You cannot be old and wise without first being young and stupid!

freebeer
Posts: 1997
Joined: Wed May 02, 2007 8:30 am
Location: Seattle area USA

Re: Your advice, ethical dilemma - job interview.

Post by freebeer » Fri May 23, 2014 7:30 pm

Gattamelata wrote:
HornedToad wrote: What you should do is go to the interview and then:
1. Point out the bias imbedded in the survey.
2. Explain what causes the bias and the extent that the bias might affect answers (5%, 10%, undeterminable amount)
3. Explain how the survey could have been better worded to get more accurate answers
4. Then answer the rest of what they asked for by analyzing the biased dataset and explaining what the results would mean etc. just like if the survey was not biased. For extra credit, analyze a range of possible results if the survey was not biased.
I like this suggestion a lot. My only suggestions is that you consider doing #4 first. They asked you an explicit question, and answering that should be your first priority...
+1. If I'm interviewing a SW developer and they are asked to code up a particular data structure or sorting algorithm for a given hypothetical problem, only after solving it the requested way is it expected to then hear "and if the hypothetical problem was real, then a better algorithm might be...". You don't want to come off as a smart-aleck or worse as making an excuse to shirk answering the requested question... instead get the extra credit.

And also - not all surveys are designed for scientific purposes to be unbiased data collection. OK if this is a job in drug studies then there's definitely an ethics issue (and watch out for a one-armed man). But if its politics, "push polling" is a tool of the trade. Some further analysis of data from such biased surveys may still be warranted (maybe the successor to "How to Lie With Statistics" is going to be "How to Lie With Big Data"). :shock:

sdrone
Posts: 108
Joined: Wed Jun 10, 2009 11:17 pm

Re: Your advice, ethical dilemma - job interview.

Post by sdrone » Fri May 23, 2014 7:34 pm

"answers were biased towards giving a positive evaluation of something that this office does"

WElcome to every internal survey every Fortune 500 company ever does. They don't want to know that a large portion of the company no longer likes their jobs; they want to know that they've positioned themselves for the future. They don't want to know that they've laid off tons of U.S. employees while hiring people offshore for less money who do a poor job; they want to know that they have a more global workforce, or that they've geographically diversified their employee intelligence.

User avatar
cfs
Posts: 4154
Joined: Fri Feb 23, 2007 1:22 am
Location: ~ Mi Propio Camino ~

Re: Your advice, ethical dilemma - job interview.

Post by cfs » Fri May 23, 2014 7:39 pm

Don't waste it

This is the perfect opportunity for you to excel. Continue with the data analysis, taking good notes on the good , the bad, and the ugly. Get ready to discuss your findings during the interview. By all means, go to that interview locked and loaded with good information, ready to provide constructive feedback on process improvement as soon as you are given the opportunity to do so. After the interview, please let us know the outcome. Good luck.

Thank you.
~ Member of the Active Retired Force since 2014 ~

HurdyGurdy
Posts: 1172
Joined: Wed May 09, 2012 10:21 pm

Re: Your advice, ethical dilemma - job interview.

Post by HurdyGurdy » Fri May 23, 2014 7:47 pm

Thank you everyone for your comments.

It is a job at a state office.

As I google my potential boss, a higher up, I find that she is committed to a "process X". "clients" have to go through "process X". She goes around making presentations about how process X is great for clients.

So the survey asks:

"I understand the importance of Process X for [my future]."
Answers are one of these two: (Yes) or (No/Not Sure).

If the person answers (No/Not sure), the survey ends that question and goes to next one.

If the person answers (Yes), the survey presents a couple of additional questions:
"My understanding of the importance of ProcessX for [my future]:"
"Before today’s appointment:"
"After today's appointment:"
With possible answers "(Poor) (Good) (Very Good) (Excellent) "

Supposedly, the question of interest is about the increase in "understanding", before and after the appointment.

However, as you can see, the survey only registers answers from those who express "I understand the importance of process X". Not many people will answer "Yes I understand the importance" and then answer "After today's appointment : my understanding is poor".

Let us say that we can assign numerical values and
Poor = 1
Good = 2
VeryGood = 3
Excellent = 4

and that we get a very simple randomized permuted situation where there is no improvement whatsoever:

Code: Select all

Person  1stChoice 2ndChoice  Difference
1          1             1               0
2          1             2               1
3          1             3               2
4          1             4              3
5          2             1              -1
6          2             2              0
7          2             3              1
8          2             4              2
9          3             1              -2
10        3             2              -1
11        3             3              0
12        3             4              1
13        4             1              -3
14        4             2              -2
15        4             3              -1
16        4             4              0
It is obvious that the mean difference is 0.
If, by virtue of the survey, we exclude ("truncate") all those who rated "poor" (1) in their second choice,

Code: Select all

Person  1stChoice 2ndChoice  Difference
2          1             2               1
3          1             3               2
4          1             4              3
6          2             2              0
7          2             3              1
8          2             4              2
10        3             2              -1
11        3             3              0
12        3             4              1
14        4             2              -2
15        4             3              -1
16        4             4              0
now the mean of the differences is 6/12 = 1/2 . Half point in the desired direction.

This higher up has a PhD from a top university.

I understand that giving, as possible answers "Poor, Good, VeryGood, Excellent", while defective (the middle anchor is between Good and VeryGood), may be just innocent bad design. But the systematic truncation, that is beyond innocent.

I left my previous field, tired of people making up stories and getting away with it.
Last edited by HurdyGurdy on Fri May 23, 2014 7:58 pm, edited 3 times in total.

123
Posts: 3893
Joined: Fri Oct 12, 2012 3:55 pm

Re: Your advice, ethical dilemma - job interview.

Post by 123 » Fri May 23, 2014 7:50 pm

Proceed with the interview including your analysis as others have suggested. From my experience this kind of a "test" is often used as an obstacle to weed out unmotivated applicants who will bail out and either cancel or not show for the interview. Many applicants will screen themselves out making the work and decision process for HR and the hiring managers that much easier. As a more mature candidate show them your stuff.
The closest helping hand is at the end of your own arm.

ABQ4804
Posts: 389
Joined: Sun Jul 24, 2011 4:08 pm

Re: Your advice, ethical dilemma - job interview.

Post by ABQ4804 » Fri May 23, 2014 8:00 pm

HurdyGurdy, the interview staff may be looking for an alternative to "Process X;" they may not all support it. Try to keep an open mind, and if you are interested in the job, need a job, and like your field of work, participate in the interview and learn more. They may need you. Good luck!

PS Don't be intimidated by the senior supervisor's PhD. A PhD does not necessarily mean someone is brilliant; they may have done extensive work/research in a specific area. An agency needs a team of specialists and obviously they have advertised they need an analyst - the job you are applying for.

User avatar
Igglesman
Posts: 436
Joined: Fri May 01, 2009 2:20 pm

Re: Your advice, ethical dilemma - job interview.

Post by Igglesman » Fri May 23, 2014 8:34 pm

What they did not teach you in school.... the final report on the results of the analysis can be written even before the data is collected. They already know what the results are going to be.

Of course it is an ethical dilemma... look at their faces in the interview when you try to question their methodology. Watch for the rolling eyeballs.

sdaehelgob
Posts: 40
Joined: Sat Jul 10, 2010 10:19 am

Re: Your advice, ethical dilemma - job interview.

Post by sdaehelgob » Fri May 23, 2014 9:12 pm

Go to the interview and interview them. Your hypothesis may pan out but to be sure you will need to collect evidence. Until then it's just a hypothesis.

HornedToad
Posts: 891
Joined: Wed May 21, 2008 12:36 am

Re: Your advice, ethical dilemma - job interview.

Post by HornedToad » Fri May 23, 2014 10:44 pm

HurdyGurdy wrote:Thank you everyone for your comments.

It is a job at a state office.

As I google my potential boss, a higher up, I find that she is committed to a "process X". "clients" have to go through "process X". She goes around making presentations about how process X is great for clients.

So the survey asks:

"I understand the importance of Process X for [my future]."
Answers are one of these two: (Yes) or (No/Not Sure).

If the person answers (No/Not sure), the survey ends that question and goes to next one.

If the person answers (Yes), the survey presents a couple of additional questions:
"My understanding of the importance of ProcessX for [my future]:"
"Before today’s appointment:"
"After today's appointment:"
With possible answers "(Poor) (Good) (Very Good) (Excellent) "

Supposedly, the question of interest is about the increase in "understanding", before and after the appointment.

However, as you can see, the survey only registers answers from those who express "I understand the importance of process X". Not many people will answer "Yes I understand the importance" and then answer "After today's appointment : my understanding is poor".

Let us say that we can assign numerical values and
Poor = 1
Good = 2
VeryGood = 3
Excellent = 4

and that we get a very simple randomized permuted situation where there is no improvement whatsoever:

Code: Select all

Person  1stChoice 2ndChoice  Difference
1          1             1               0
2          1             2               1
3          1             3               2
4          1             4              3
5          2             1              -1
6          2             2              0
7          2             3              1
8          2             4              2
9          3             1              -2
10        3             2              -1
11        3             3              0
12        3             4              1
13        4             1              -3
14        4             2              -2
15        4             3              -1
16        4             4              0
It is obvious that the mean difference is 0.
If, by virtue of the survey, we exclude ("truncate") all those who rated "poor" (1) in their second choice,

Code: Select all

Person  1stChoice 2ndChoice  Difference
2          1             2               1
3          1             3               2
4          1             4              3
6          2             2              0
7          2             3              1
8          2             4              2
10        3             2              -1
11        3             3              0
12        3             4              1
14        4             2              -2
15        4             3              -1
16        4             4              0
now the mean of the differences is 6/12 = 1/2 . Half point in the desired direction.

This higher up has a PhD from a top university.

I understand that giving, as possible answers "Poor, Good, VeryGood, Excellent", while defective (the middle anchor is between Good and VeryGood), may be just innocent bad design. But the systematic truncation, that is beyond innocent.

I left my previous field, tired of people making up stories and getting away with it.
Honestly, I think you are reading too much into the question/being too academic about it. The way the survey is written, it answers the following question:

For those who care about process X, what was their understanding of process X before and after this appointment. i.e. how good of a job did the person do to explain something to someone who is interested in process X. That's a reasonable metric.

It basically assumes that if the person is not interested in process X, then how good of a job someone did explaining process X to the person is irrelevant because they are not interested in it so they shouldn't be measured on it.

Depending on the precise nature of the question being asked, you could structure the survey in a thousand different directions. This is not so ridiculous that I would consider it an ethical dilemma, especially considering that you don't know what precise question they cared most of figuring out in the survey.....

Edit: I also think it's mistaken to assume people won't say I understand the importance, but my understanding of it is poor. There's lots of areas that I understand are important ideas/theories/proposals, but the mechanics of how it works and what makes it work I have no idea.

HurdyGurdy
Posts: 1172
Joined: Wed May 09, 2012 10:21 pm

Re: Your advice, ethical dilemma - job interview.

Post by HurdyGurdy » Wed May 28, 2014 9:07 am

Thank you everybody for their very helpful comments. As an update, I went to the interview and tried to do what most of you suggested: did the presentation, added my observations on the data due to the design of the survey, and cautioned about the interpretation. All was done in a very cordial manner.

Lesson learned and applied!

WorkToLive
Posts: 617
Joined: Thu Nov 12, 2009 2:19 pm

Re: Your advice, ethical dilemma - job interview.

Post by WorkToLive » Wed May 28, 2014 9:10 am

Sounds like you handled it perfectly. Good luck on the outcome! Keep us posted.

User avatar
JupiterJones
Posts: 2701
Joined: Tue Aug 24, 2010 3:25 pm
Location: Nashville, TN

Re: Your advice, ethical dilemma - job interview.

Post by JupiterJones » Wed May 28, 2014 11:21 am

I don't see a problem with how you handled it at all.

An open discussion of potential biases, confounding factors, etc., should be part of any legitimate data analysis.

If they don't hire you because you did exactly what a professional analyst should do, then I guess that tells you something about them. Interviews go both ways, you know.
Stay on target...

Elena78
Posts: 90
Joined: Thu Jul 19, 2012 3:58 pm

Re: Your advice, ethical dilemma - job interview.

Post by Elena78 » Wed May 28, 2014 2:43 pm

Gattamelata wrote:
HornedToad wrote: What you should do is go to the interview and then:
1. Point out the bias imbedded in the survey.
2. Explain what causes the bias and the extent that the bias might affect answers (5%, 10%, undeterminable amount)
3. Explain how the survey could have been better worded to get more accurate answers
4. Then answer the rest of what they asked for by analyzing the biased dataset and explaining what the results would mean etc. just like if the survey was not biased. For extra credit, analyze a range of possible results if the survey was not biased.
I like this suggestion a lot. My only suggestions is that you consider doing #4 first. They asked you an explicit question, and answering that should be your first priority. Doing otherwise suggests that you think you know better than they do how the interview should go. It might mean preparing two assessments: one that ignores the bias in the survey, and one that includes the bias in the assessment. In fact, that could be a grand slam interview presentation: "Here's my assessment of the data as presented. But after getting more information on the survey, I found significant bias. This is my assessment with the bias factored in. Here are the differences, and this shows the impact of the bias. Here's how I'd redesign the survey to remove the bias and increase accuracy." That way the arc of your interview starts with meeting their expectations, then moves to surprising them, then exceeding their expectations, and finally giving them something actionable. I think it might be hard not to hire that!
Great suggestions here. I also think this is a good chance for you to interview them. If they are defensive about the feedback on the survey it might not be a good fit for a purest like yourself. If they think it's a great observation, you will have scored points with them and them with you.

Whoops - just saw you updated .... I guess you got your response!

Post Reply