4 Literature review and citations/references

Figure 4.1: Literature reviews and references

Your may have done a literature survey as part of your proposal. This will be incorporated into your dissertation, not left as separate stand-alone. Most economics papers include a literature review section, which may be a separate section, or incorporated into the paper’s introduction. (See organising for a standard format.)

Some disambiguation: literature survey/review paper vs literature survey section (unfold…)

Some disambiguation: literature survey/review paper vs literature survey section

A ‘literature survey’ paper: Some academic papers are called ‘literature surveys’ or ‘literature reviews’. These try to summarise and discuss the existing work that has been done on a particular topic, and can be very useful. See, for example, works in The Journal of Economic Perspectives, the Journal of Economic Literature, the “Handbook of [XXX] Economics”

Many student projects and undergraduate dissertations are mainly literature surveys.

4.1 What is the point of a literature survey?

Your literature review should explain:

what has been done already to address your topic and related questions, putting your work in perspective,
what techniques others have used, what are their strengths and weaknesses, and how might they be relevant tools for your own analysis,
and what is the consensus of the evidence,

Take notes on this as you read, and write them up!

4.2 What previous work is relevant?

Focus on literature that is relevant to your topic only.

But do not focus only on articles about your exact topic! For example, if your paper is about the relative price of cars in the UK, you might cite papers (i) about the global automobile market, (ii) about the theory and evidence on competition in markets with similar features and (iii) using econometric techniques such as “hedonic regression” to estimate “price premia” in other markets and in other countries.

Consider: If you were a doctor in Colchester and wanted to know whether a medicine would be effective for your patients, would you only consider medical studies that ran tests on Colchester residents, or would you consider more general national and international investigations?

4.3 What are “good” economics journal articles?

You should aim to read and cite peer-reviewed articles in reputable economics journals. (Journals in other fields such as Finance, Marketing and Political Science may also be useful.) These papers have a certain credibility as they have been checked by several referees and one or more editors before being published. (In fact, the publication process in Economics is extremely lengthy and difficult.)

Which journals are “reputable”? Economists spend a lot of time thinking about how to rank and compare journals (there are so many papers written about this topic that they someone could start a “Journal of Ranking Economics Journals”. For example, “REPEC” has one ranking, and SCIMAGO/SCOPUS has another one. You may want to focus on journals ranked in the top 100 or top 200 of these rankings. If you find a very interesting and relevant paper published somewhere that is ranked below this, is okay to cite it, but you may want to be a bit more skeptical of its findings.

Any journal you find on JSTOR is respectable, and if you look in the back of your textbooks, there will be references to articles in journals, most of which are decent.

Unpublished (or not-yet-published) “working papers”

You may also find unpublished “working papers” or ‘mimeos’; these may also be useful as references. In fact, the publication process in Economics is so slow (six years from first working paper to publication is not uncommon) that not consulting working papers often means not being current.

However, it is more difficult to evaluate the credibility of this ‘grey literature’, as they have not been through a process of peer review. Nonetheless, if the author has published well and has a good reputation, it might be more likely that these are worth reading and citing. Some working paper series are vetted, such as NBER; in terms of credibility, these might be seen as something in between a working paper and a publication.

Which of the following are “peer-reviewed articles in reputable economics journals”? Which of the following may be appropriate to cite in your literature review and in your final project?

Klein, G, J. (2011) “Cartel Destabilization and Leniency Programs – Empirical Evidence.” ZEW - Centre for European Economic Research Discussion Paper No. 10-107
Spencer, B. and Brander, J.A. (1983) “International R&D Rivalry and Industrial Strategy”, Review of Economic Studies Vol. 50, 707-722
Troisi, Jordan D., Andrew N. Christopher, and Pam Marek. “Materialism and money spending disposition as predictors of economic and personality variables.” North American Journal of Psychology 8.3 (2006): 421.
The Economist,. ‘Good, Bad And Ugly’. Web. 11 Apr. 2015. [accessed on…]
Mecaj, Arjola, and María Isabel González Bravo. “CSR Actions and Financial Distress: Do Firms Change Their CSR Behavior When Signals of Financial Distress Are Identified?.” Modern Economy 2014 (2014).
Universities, U. K. “Creating Prosperity: the role of higher education in driving the UK’s creative economy.” London Universities UK (2010).

Answer (unfold):

Only number 2 is a ‘peer reviewed article in a reputable economics journal’. All of these might be useful to cite, however, for various reasons.

4.4 How to find and access articles

You should be able to find and access all the relevant articles online. Leafing through bound volumes and photocopying should not be needed. (Having been a student in the late 90’s and 2000’s, I wish I could get those hours back.)

Figure 4.2: The old way!

Good online tools include Jstor (jstor.org) and Google Scholar (scholar.google.co.uk). Your university should have access to Jstor, and Google is accessible to all (although the linked articles may require special access). You will usually have the ‘most access’ when logged into your university or library computing system. If you cannot access a paper, you may want to consult a reference librarian.

It is also OK, if you cannot access the journal article itself, to use the last working paper version (on Google scholar find this in the tab that says “all X versions”, where X is some number, and look for a PDF). However, authors do not always put up the most polished versions, although they should do to promote open-access. As a very last resort, you can e-mail the author and ask him or her to send you the paper.

When looking for references, try to find ones published in respected refereed economics journals (see above).

4.5 Good starting points: Survey article, course notes, and textbooks

A “survey article” is a good place to start; this is a paper that is largely a categorization and discussion of previous work on a particular topic. You can often find such papers in journals such as

the Journal of Economic Perspectives,
the Journal of Economic Surveys,
and the Journal of Economic Literature.

These will be useful as a “catalog” of papers to read and considers citing. They are also typically very readable and offer a decent introduction to the issue or the field.

It is also helpful to consult module (course) notes and syllabi from the relevant field. Do not only limit yourself to the ones at your own university; many of universities make their course materials publicly accessible online. These will not only typically contain reading lists with well-respected and useful references, they may also contain slides and other material that will help you better understand your topic and the relevant issues.

However, be careful not to take material from course notes without properly citing it. (Better yet, try to find the original paper that the course notes are referring to.)

Textbooks serve as another extremely useful jumping off point. Look through your own textbooks and other textbooks in the right fields. Textbooks draw from, and cite a range of relevant articles and papers. (You may also want to go back to textbooks when you are finding the articles you are reading too difficult. Textbooks may present a simpler version of the material presented in an article, and explain the concepts better.)

4.6 Backwards and forwards with references

The biggest challenge is often to find the first relevant article When you find a relevant article you have a great starting point, which may serve as a gateway to much more. Go ahead and look for its “family.” You may want to go back to earlier, more fundamental references, by looking at the articles that this paper cited. See what is listed as “keywords” (these are usually given at the top of the paper), and “JEL codes”. Check what papers this paper cites, and check what other papers cited this paper. On Google scholar you can follow this with a link “Cited by…” below the listed article. “Related articles” is also a useful link.

4.7 Citations

Students at Exeter (and beyond) may find the University of Exeter Business School’s "A Guide to Citing, Referencing and Avoiding Plagiarism very useful (Truelove, 2014).

This guide offers tips on standard ways to cite a wide range of material, including, e.g., YouTube videos.

You may find some slight differences between the guide linked above (Truelove, 2014) and the points I make in the present web book. However, where the suggestions differ (e.g., I do not emphasise citing page numbers in articles), this is unlikely to matter in a way that affects your mark and the quality of your work.

A common citation style is ‘author-date’. Here we may refer to the papers’ authors and surnames followed by the publication dates, e.g.,

There are some other rules too, e.g., when there are more than two authors you list only the first authors surname plus the Latin ‘et al.’

Time travel is frequently espoused by prominent authors and scientists as well as laymen (Wells, 1895; Brown, 1985; Preston and Logan, 1989). However, it has lead to documented significant negative consequences on at least one occasion (Sanchez, 2018).

If it is already clear which author/paper is referred to, you only need to put the publication date in parentheses…

Logan (1989) argued that we should maintain excellence in our interpersonal relations.

With a direct quote…

Ted Logan (1989, p. 35) advocated that people should “be excellent to each other”.

If you are using a direct quote, do you need to include the page number for that quote? This is standard practice in many fields, and it is recommended in the Exeter Guide, and in many style manuals. However, I don’t always do it myself and it is not universal in Economics journals. E.g., the Journal of Economic Perspectives does it, but the Review of Economic Studies does not. In my opinion, it is a bit obsolete: it is not convenient when considering works that may be hosted online both in html and pdf formats.

Still, even though I don’t like it so much, you probably should include these page numbers along with direct quotes, unless advised otherwise.

Keep track of all references and citations

You may find it helpful to use software to help you manage your citations

Technology and tools keep changing, of course. These tools were relevant as of 2019, but there may be better ones by the time you read this book.

A storage “database” of citations (e.g., Jabref, Zotero, Endnote, Mendeley); these interface well with Google Scholar and Jstor
An automatic “insert citation” and “insert bibliography” in your word processing software

Use a tool like Endnote to manage and insert the bibliographies, or use a bibliography manager software such as Zotero or Jabref.

Further discussion: Citation management tools

Citation and bibliography styles

There are a number of citation formats used by various fields and journals, as you can see HERE. In my opinion the precise style you choose doesn’t matter too much. Just choose some standard style and try to be consistent with it (and using citation and referencing tools can help with this).

At the risk of repeating myself, the most important things are:

to make it is clear to the reader where everything came from (each precise statement or idea), so that she can correctly assume that what remains is your own original work
all quoted material must be put in quotes-marks (inverted commas or offset)
making sure it is easy for the reader to look up and track down each reference

List of works cited

Put your list of references in alphabetical order by author’s last name (surname).

Include all articles and works that you cite in your paper; do not include any that you don’t cite.

Avoiding plagiarism and academic offenses**

Here is a definition of plagiarism

The main point is that you need to cite everything that is not your own work. Furthermore, be clear to distinguish what is your own work and your own language and what is from somewhere/someone else.

Why cite? Not just to give credit to others but to make it clear that the remaining uncited content is your own.

Here are some basic rules:¹

Do not submit anything that is not your own work.
Never copy from friends.
Do not copy your own work or previously submitted work. (Caveat: If you are submitting a draft or a ‘literature review and project plan’ at an earlier stage, this can be incorporated into your final submission.
Don’t copy text directly into your work, unless:
- you put all passages in quotation marks: beginning with ’ and ending with ’, or clearly offset from the main text
- you cite the source of this text.

It is not sufficient merely to add a citation for the source of copied material following the copied material (typically the end of a paragraph). You must include the copied material in quotation marks. … Ignorance … is no defence.’ (ibid, pp. 15 )

(‘Ibid’ means ‘same as the previous citation’.)

WARNINGS

Your university may use sophisticated plagiarism-detection software. Markers may also report if the paper looks suspect

Before final submission, they may ask you to go over your draft and sign that you understand the contents and you have demonstrated that the work is your own.

Not being in touch with your supervisor may put you under suspicion.

Your university may give a Viva Voce oral exam if your work is under suspicion. It is a cool-sounding word but probably something you want to avoid.

Your university may store your work in its own database, and can pursue disciplinary action, even after you have graduated.

Penalties may be severe, including failure with no opportunity to retake the module (course). You may even risk your degree!

Comprehension questions (answers in folded box below)

Q-A. True or false: “If you do not directly quote a paper you do not need to cite it”

Q-B. You should read and cite a paper (choose all that are correct)…

If it motivates ‘why your question is interesting’ and how it can be modeled economically
Only if it asks the same question as your paper
Only if it is dealing with the same country/industry/etc as you are addressing
If it has any connection to your topic, question, or related matters
If it answers a similar question as your paper
If it uses and discusses techniques that inform those you are using

Q-A. False. You need to cite any content and ideas that are not your own.

Q-B. Answers: 1, 5, and 6. Note that 2 and 3 are too narrow criteria, and 4 is too broad.]

4.8 How to write about previous authors’ analysis and findings

Use the right terminology

Oops (another counter-example):

“Johnson et al. (2000) provides an analytical framework that sheds substantial doubt on that belief. When trying to obtain a correlation between institutional efficiency and wealth per capita, they are left with largely inconclusive results.”

They are not trying to “obtain a correlation”; they are trying to measure the relationship and test hypotheses.

“Findings”: Critically examine sources

Don’t take everything that is in print (or written online) as gospel truth. Be skeptical and carefully evaluate the arguments and evidence presented. Try to really survey what has been written, to consider the range of opinions and the preponderance of the evidence. You also need to be careful to distinguish between “real research” and propaganda or press releases.

The returns to higher education in Atlantis are extremely high. For the majority of Atlanian students a university degree has increased their lifetime income by over 50%, as reported in the “Benefits of Higher Education” report put out by the Association of Atlantian Universities (2016).

But don’t be harsh without explanation:

Smith (2014) found a return to education in Atlantis exceeding 50%. This result is unlikely to be true because the study was not a very good one.

“Findings:” “They Proved”

“Proved” is usually not the correct word to use other than when considering a mathematical proof (including in economic theory) derived from assumptions (and even here one needs to be very precise). It is usually the wrong word choice in the context of empirical analysis or for statements pertaining to the real world rather than to theory. Typically it is better to write something along the lines of “offered evidence suggesting”.

A theoretical economic model can not really prove anything about the real world; they typically rely on strong simplifying assumptions.

Cringe:

Through their economic model, they prove that as long as elites have incentives to invest in de facto power, through lobbying or corruption for example, they will invest as much as possible in order to gain favourable conditions in the future for their businesses.

Better:

In their two period model, which assumes [details of key assumptions here], they find that when an elite Agent has an incentive to invest in de facto power, he invests a strictly positive amount, up to the point where marginal benefit equals marginal cost”

Empirical work does not “prove” anything (nor does it claim to).

It relies on statistical inference under specific assumptions, and an intuitive sense that evidence from one situation is likely to apply to other situations.

Shudder:

“As Smith et al (1999) proved using data from the 1910-1920 Scandanavian stock exchange, equity prices always increase in response to reductions in corporate tax rates.”

Better:

“Smith et al (1999) estimated a VAR regression for a dynamic CAP model using data from the 1910-1920 Scandanavian stock exchange. They found a strongly statistically significant negative coefficient on corporate tax rates. This suggests that such taxes may have a negative effect on publicly traded securities. However, as their data was from a limited period with several simultaneous changes in policy, and their results are not robust to [something here], further evidence is needed on this question.”

Use the language of classical statistics:

Of course, you may also have been taught, or may consider pursuing the Bayesian (rather than classical or frequentist) statistical approach.

Hypothesis testing, statistical significance, robustness checks, magnitudes of effects, confidence intervals.

Note that generalisation outside the data depends on an intuitive sense that evidence from one situation is likely to apply to other situations.

“Findings”: How do you (or the cited paper) claim to identify a causal relationship?

This policy was explained by Smith and Johnson (2002) in their research on subsidies and redistribution in higher education. Their results showed that people with higher degrees have higher salaries and so pay higher taxes. Thus subsidizing higher education leads to a large social gain.

The results the student discusses seem to show an association between higher degrees and higher salaries. The student seems to imply that the education itself led to higher salaries. This has not been shown by the cited paper. Perhaps people who were able to get into higher education would earn higher salaries anyway. There are ways economists used to try to identify a “causal effect” (by the way, this widely used term is redundant as all effects must have a cause), but a mere association between two variables is not enough

As inflation was systematically lower during periods of recession, we see that too low a level of inflation increases unemployment.

Economists have long debated the nature of this “Phillips curve” relationship. There is much work trying to determine whether the association (to the extent it exists) is a causal one. We could not rule out reverse causality, or third factor that might cause changes in both variables.

4.8.1 …Stating empirical results

Don’t write: “I accept the null hypothesis.”

Do write: “The results fail to reject the null hypothesis, in spite of a large sample size and an estimate with small standard errors” (if this is the case).

Note: The question of what to infer from acceptance/rejection of null hypotheses is a complex difficult one in Classical (as opposed to Bayesian) statistics. This difficulty is in part philosophical: classical hypothesis testing is deductive, while inference is necessarily inductive.

4.8.2 What to report

A frequent comment:

You need to read this paper more clearly; it is not clear what they conclude nor what their evidence is.

It is not enough to just state the results that the authors claim. Make it clear what they mean, how they get to this point, and what sort of evidence or argumentaition they bring.

4.9 Organising your literature review

A common marking comment:

These papers seem to be discussed in random order – you need some structure organising these papers thematically, by finding, by technique, or chronologically perhaps.

How should you organise it? In what order?

Thematically (usually better)
By method, by theoretical framework, by results or assumptions, by field
Chronologically (perhaps within themes)

Exercise: Compare how the literature review section is organized in papers you are reading.

Figure 4.3: Organising a set of references

Q: What sort of structure am I using in this outline?

It may also be helpful to make a ‘table’ of the relevant literature, as in the figure below. This will help you get a sense of the methods and results, and how the papers relate, and how to assess the evidence. You may end up putting this in the actual paper.

Figure 4.4: Organisational table from Reinstein and Riener, 2012b

4.10 What if you have trouble reading and understanding a paper?

Consult a survey paper, textbook, or lecture notes that discuss this paper and this topic
Try to find an easier related paper
Ask your supervisor for help; if he or she can
Try to understand what you can; do not try to “fake it”

“Faking it”: pretending that you understand a topic when you do not. I think the perceived need to pretend to understand things one does not happens at all levels od society and education. It is very harmful (see the fable ‘the Emperor wears no clothes’); it is much better to admit to one’s limited understanding, and to try to learn more

4.11 Some literature survey do’s and don’ts

Do not cite irrelevant literature.

Do not merely list all the papers you could find.

Discuss them, and their relevance to your paper.
What are their strengths and weaknesses? What techniques do they use, and what assumptions do they rely on? How do they relate to each other?

Use correct citation formats.

Try to find original sources (don’t just cite a web link).

Don’t just cut and paste from other sources. And make sure to attribute every source and every quote. Be clear: which part of your paper is your own work and what is cited from others? The penalties for plagiarism can be severe!

Critically examine the sources, arguments, and methods

4.12 Comprehension questions: literature review

How to discuss empirical results: “Causal” estimation, e.g., with Instrumental Variables

Which is the best way to state it?

“As I prove in table 2, more lawyers lead to slower growth (as demonstrated by the regression analysis evidence).”
“Table 2 provides evidence that a high share of lawyers in a city’s population leads to slower growth.”
“Table 2 shows that a high share of lawyers in a city’s population is correlated with slower growth.”

Answer: (unfold)

The second one; if this is really causal evidence.

Which is better stated?

“However, when a set of observable determinants of city growth (such as Census Region growth) are accounted for, the estimate of this effect becomes less precise.”

“In the correct regression I control for all determinants of city growth and find that there is no effect of lawyers on growth”

Answer: (unfold)

The first one. There is no ‘correct regression’. It is also not really correct in classical statistics to ‘find no effect’.

Stating empirical results: descriptive

If the results are descriptive, don’t characterize them as causal!

Don’t write

“I find a significantly lower level of inflation during periods of recession, and the difference is economically meaningful. This relationship is statistically significant and the data is accurately measured. Thus I find that inflation increases unemployment.”

Do write:

“Using the US data from 1850-1950, I find that inflation is lower during periods of recession. This is statistically significant in a t-test [or whatever test] at the 99% level, and the difference is economically meaningful. This is consistent with the theory of …, which predicts that lower inflation increases unemployment. However, other explanations are possible, including reverse causality, and unmeasured covarying lags and trends.”

Some tips on writing a good paper– relevant to literature reviews²

Answer the question
Provide clear structure and signposting
Demonstrate an ability for critical analysis
Refer to your sources
Produce a coherent, clear argument
Take time to proofread for style and expresssion

Rephrased from University of Essex material, as seen in Department of Economics, EC100 Economics for Business Handbook 2017-18, https://www1.essex.ac.uk/economics/documents/EC100-Booklet_2017.pdf accessed on 20 July 2019, pp. 15-16↩︎
- Source “Assignment Writing Skills Essex Business School 3rd year 2012”↩︎