Questions for the course "Scientific Methods" with all questions in one.
Please know, that it is not intended, that you should know everything which are in this quiz, at the exam.
The content of this quiz, is based on nearly all slides.
Click the button to start the quiz
Criteria for research activity
Novel/New
Creative
Uncertain
Systematic
Transferable/Reproducible
Research process steps
Define the topic
Search / Investigate
Discover
Communicate / Disseminate
What are the Philosophical worldviews?
Positivism
Constructivism
Transformative
Pragmatic
What does Research design include?
Quantitative
Qualitative
Mixed methods
What Research Methods has been introduced?
Questions/Hypotheses
Data collection
Data analysis
Interpretation
Validation
Why should you know about philosophical worldviews?
Support argument for chosen approach and methods
What are the key characteristics of positivism and post-positivism in scientific research?
Positivism is more prevalent in quantitative research and starts with a theory to test
Post-positivists focus on refuting theories rather than proving them
What is a key assumption of positivism in scientific research?
Objectivity is key
How do post-positivists view the concept of truth?
There is no absolute truth
What is the main focus of post-positivist research?
Refuting theories rather than proving them
Which research method is most closely associated with positivism?
Controlled experiment
survey research
case studies
Why is reductionism important in positivist research (especially in software engineering)?
It allows phenomena to be studied in isolation from their context
What is a key assumption of constructivism in scientific research?
Knowledge cannot be separated from its human contexts (experience and environment).
In which type of research is constructivism more prevalent?
Qualitative research.
What is the main focus of constructivist research?
Understanding how people make sense of the world.
How does theory typically emerge in constructivist research?
Inductively, tied to the contexts.
How are research findings viewed in constructivist research?
As an interpretation shaped by the researcher’s own experience and background.
What is the origin of the transformative worldview in research?
It emerged from a political movement to represent and improve conditions for minorities, e.g., feminist studies.
What do transformative researchers believe about the role of research?
Research needs to be intertwined with making changes in the real world.
How is the transformative worldview applied in software engineering?
It includes research that actively seeks to challenge existing perceptions about software practice, such as the open source movement and the agile community.
Which research method most closely reflects the philosophy of critical theorists in the transformative worldview?
Action research.
What type of studies do critical theorists often use to draw attention to things that need changing?
Case studies.
What is a key acknowledgment of the pragmatic worldview in research?
Knowledge is incomplete.
How does the pragmatic worldview view knowledge?
Knowledge has a degree of relativism - what is useful for me is not necessarily useful for you.
What do pragmatic researchers emphasize in their research?
The research problem and use all approaches available to understand the problem.
What methods do pragmatists typically use in their research?
Whatever methods that would work, usually mixed methods.
What approach does Quantitive research focus on?
Testing objective theories by examining relationships between variables
What approach does Qualitative research focus on?
Understanding social phenomena from the perspective of the people being studied
What approach does Mixed methods research focus on?
Using both quantitative and qualitative research methods to understand a research problem
What philosophical worldview is most closely associated with qualitative research?
Constructivism
Transformative
What philosophical worldview is most closely associated with quantitative research?
Positivism
What philosophical worldview is most closely associated with mixed methods research?
Pragmatic
What strategy of inquiry is most closely associated with qualitative research?
Ethnography
Exploratory case study
Qualitative survey
Interviews
What strategy of inquiry is most closely associated with quantitative research?
Experiments
Survery
What strategy of inquiry is most closely associated with mixed methods research?
Convergant parallel
Explanatory sequential
Exploratory sequential
What methods are most closely associated with qualitative research?
Open-enden questions
Text or image data
Flexible inquiry
What methods are most closely associated with quantitative research?
Closed-ended questions
Numeric data
Pre-determined
What methods are most closely associated with mixed methods research?
Both open and closed-ended questions
Both text and numeric data
What practices are most closely associated with qualitative research?
Validates accuracy of the data (triangulate)
Makes interpretation of the data
Collaborate with participants
Study the contexts or setting of the participants
Brings personal value into the study
What practices are most closely associated with quantitative research?
Test or verifies theories
Identify variables to study
Relates variables in questions or hypotheses
Uses unbiased approaches
Use statistical procedures
What practices are most closely associated with mixed methods research?
Collects both qualitative and quantitative data
Integrates data at different stages of inquiry
Why do literature review?
Provides the readers the results of other related studies (Knowledge of the topic)
Position your own work with respect to others works (doing something new or adding to a contribution?)
Motivates the importance of your own work (Current gap in the literature)
How to use literature (Qualitative studies) as suggested by the teacher?
Points out gap in the results or findings of existing work
How to use literature (Quantitative studies) as suggested by the teacher?
Introduce a theory to be examined
To provide direction for the research questions and hypothesis
What is a theory?
A supposition or a system of ideas intended to explain something, especially one based on general principles independent of the thing to be explained
What is examples of theories?
Einstein's general relativity
Darwin's theory of evolution
What is the steps brought forward, for quantitative research?
Researcher tests ro verifies a theory
Researcher tests hypotheses or research questions from theory
Researcher measures or observes variables derived from theory
Researcher measures or observes variables using an instrument to obtain scores
What is the steps brought forward, for qualitative research?
Researcher gathers information (eg. interviews, observations, surveys)
Researcher asks open-ended questions
Researcher analyses data to form themes and categories
Researcher looks for a broad patterns, generalisations, or theories from themes and categories
Researcher poses generalisation or theories from past observations
What approach does quantitative research take?
Deductive
What approach does qualitative research take?
Inductive
What way does deductive reasoning work?
Down -> From a theory towards observations and measurements
What way does inductive reasoning work?
Up -> From observations and measurements towards a theory
What type of reasoning is used in mixed methods research?
Both deductive and inductive
What is a research problem?
A clear expression about an area of concers
A condition to be improved upon
A difficulty to be eliminated
A troubling question that exists in literature
What is a research question accordingly to bryman 2007?
Does not state how to do something, offers a vague or borad proposition, or presents a value question
How to identify a research problem (source)?
From the literature
Personal observations
Collective observations
What could a research problem be?
A gap in the literature
A conflicting result in the literature
A need to put attention to a (under-studied) topic
Where does research questions usually come from?
Expert advice
Literature & experience
Research problem
Research intent or purpose
What are some typical characteristics of good research questions?
Clear
Unambiguous
Show the Purpose of the research
Answerable
Not trivial
What category of research design does the following type of questions belong to? "Does X exist?", "How does X differ from Y?", "How does X normally work?"
qualitative
What category of research design does the following type of questions belong to? "How often does X occur?", "Do occurrences of X correlate with the occurences of Y?", "Does X cause more Y than does Z?"
Quantitative
What is a hypothesis?
Predictions, the researcher makes about the expected outcomes of relationships among variables
What to forms does a hypothesis take?
Null hypothesis
Alternative/directional hypothesis
What is a null hypothesis?
Makes a prediction that no relationship or no significant difference exists between groups on a variable
What is an alternative/directional hypothesis?
Makes a prediction (higher, more , lower, less, etc) about the expected outcome
What does RQ stand for?
Research question
Whoes does HO stand for?
Hypothesis
What type of hypothesis is the example research question (RQ) and hypothesis (HO): "RQ: Are teens better at math than adults? HO: Age as no effect on mathematical ability?"
Null hypothesis
What type of hypothesis is the example hypothesis (HO): "HO: People with a a high exposure to ultravielt light will have a higher frequency of skin cancer."
Alternative/directional hypothesis
What is the independent variable in the RQ: "Does the use of pair programming improve software quality compared to individual programming?"
Type of programming (pair vs individual)
What is the dependent variable in the RQ: "Does the use of pair programming improve software quality compared to individual programming?"
Software quality
What is the null hypothesis (H0) in the RQ: "Does the use of pair programming improve software quality compared to individual programming?"
There is no significant difference in software quality between projects using pair programming and projects using individual programming
What is the alternative hypothesis (H1) in the RQ: "Does the use of pair programming improve software quality compared to individual programming?"
Software projects using pair programming will have significantly higher quality than projects using individual programming
RQ: "How does the amount of time spent studying influence test score?". Identify variables
A researcher randomly assigns college students to a prescribed amount of study time. She plans to examine the extent to which study time leads to improved test scores.
Independent variable: Amount of time spent studying
Dependent variable: Test score
RQ: "How does the amount of time spent studying influence test score?". Formulate null and alternative hypothesis
A researcher randomly assigns college students to a prescribed amount of study time. She plans to examine the extent to which study time leads to improved test scores.
Null hypothesis: Study time has no effect on test scores
Alternative hypothesis: The students who spent more time studying will have higher test scores
RQ: "Does increasing the number of code reviews in a software development project have a significant effect on reducing the number of post-release defects??". Identify variables
A researcher aims to investigate the relationship between the number of code reviews conducted during the software development process and the subsequent number of post-release defects scores.
Independent variable: No. of code reviews
Dependent variable: No. of post-release defects
RQ: "Does increasing the number of code reviews in a software development project have a significant effect on reducing the number of post-release defects??". Formulate null and alternative hypothesis
A researcher aims to investigate the relationship between the number of code reviews conducted during the software development process and the subsequent number of post-release defects scores.
Null hypothesis: Number of code reviews in a software development project has no effect on the number of post-release defects
Alternative hypothesis: Increasing the number of code reviews in a software development project will lead to a decrease in the number of post-release defects
What defines the researchs method; Survey
Identify the characteristics of a broad population of individuals
Could be qualitative (exploratory w / open questions) or quantitative (confirmatory w / closed questions)
Questionnaires (web-based, telephone, paper-based, interviews)
Research question is about the nature of a particular target population (e.g., SW developers working with Agile in Denmark)
What defines the research method; Ethnography
Field Observation
Research question focuses on the cultural practices of a particular community
Study how developers (and maybe other roles) work and make sense of their work in their natural settings
What defines the research method; Case study
Offer in-depth understanding of how and why certain phenomena occur, and can reveal the mechanisms by which cause-effect relationships occur
Could be: exploratory (inductive) or confirmatory (deductive)
Research question focuses on “how”or “why”something occurs
What defines the research method; Experiment
A controlled investigation of two or more variables
Need to determine independent and dependent variables
Research questions formulated to understand correlation or causality
More often hypotheses are formulated
What defines the research method; Mixed methods (Convergent parallel)
Qualitative and quantitative data collection and analysis done in parallel
What defines the research method; Mixed methods (Sequential)
Collect and analyse quantitative data ->collect and analyse qualitative data (Explanatory)
Collect and analyse qualitative data -> collect and analyse quantitative data (Exploratory)
What are research ethics?
Set of ethics that govern how scientific research is performed at research institutions and how is is disseminated
What is a typical and historical practice of research, that is now considered unethical?
Torture or inhumane treatment of research subjects
Milgram experiment of 1961
Principles of research integrity
Honesty
Transparency
Accountability
In the principle of research integrity, what does honesty mean?
Ensuring trustworthiness of research
Requires accurate reporting
What areas are honesty important in?
Presenting, intereting research results
Making claims based on findings
Review research
Acknowledging the work of others (Gift & Ghost authorship)
In the principle of research integrity, what does transarency mean?
Ensures credibility of research
Requires openness
What areas are transparency important in?
Planning research
Applied research methods
Results and conclusions
Conflict of interest
In the principle of research integrity, what does accountability mean?
Ensure all parties involved in the research should be accountable
Researchers need to accept responsibility
What areas are accountability important in?
Accuracy and reliability of results
Adherence to regulations and guidelines
Taking appropriate action in case of misconduct
What regulation are researchers at SDU bound by?
SDU's Review Ethics Committee
The Danish Code of Conduct for Research Integrity
What are typical types of scientific frauds?
Fabrication
Falsification
Plagiarism
Fake peer review
What does scientific fraud, regarding fabrication, mean?
Making up data or results and recording or reporting them
"Filling out" remaining experiment runs
Claims made on incomplete data or assumed results
What does scientific fraud, regarding falsification, mean?
Changing research results (data)
Omitting research results (data)
Manipulation of research materials, equipment, or processes
What does scientific fraud, regarding plagiarism, mean?
Using or representing the work of others as your own work
Citing or using not yet published work
Citing or using not publicly available work
Forgetting to cite
Is using or representing the work of others as your own work, constitutes plagiarism if it is done unintentionally?
Yes
Is the following plagiarism? if so, why?
Original source (Nickerson, 1999):
"In order to communicate effectively with other people, one must have a reasonably accurate idea of what they do and do not know that is pertinent to the communication. Treating people as though they have knowledge that theydo not have can result in miscommunication and perhaps embarrassment."
The work to be reviewed (author, unknown):
"Effective communication depends on a generally accurate knowledge of what the audience knows. If a speaker assumes too much knowledge about the subject, the audience will either misunderstandor be bewildered."
Yes, as paraphrasing still needs to be cited
What are typical types of self-plagiarism?
Using a dataset from a previous study, without making the reader aware
Submitting work for publication, containing conclusions or passages that have been published before, without citing the original source
Publishing similar or identical work in multiple publications / journals
Is the following plagiarism? if so, why?
Article 1 (Krashen, S., 2012):
"Sometimes we have to write long papers, but most of the time, it is unnecessary: the papers often contain long introductions more suitable for doctoral dissertations or review “state of the art” papers apparently designed to provide evidence that the author is well-read."
Article 2 (Krashen, S., 2013):
"Sometimes we have to write long papers to treat a topic adequately, but much of the time, it’s unnecessary: Papers often contain long introductions more suitable for doctoral dissertations, apparently intended to provide evidence that the author is well-read..."
Yes, this is self-plagiarism
Yes, as article 1 is not cited in article 2
Is it allowed to use graphs, charts, figures or images?
Yes, but they must be cited in the caption
Yes, however tables should not be duplicate, it should be a recreation of the data
No, if they are not cited
What is the responsibilities as a researcher?
You must not inflict harm
Do not cause physical or psychological pain
Maintain privacy and confidentiality
Be cautious where data are stored in regards to privacy
You must not deceive
Honest to the participants
Honest to the audience
No laws are broken
In which order, should authorship be given in this case?
"Jamal is a graduate student working under the supervision of professor, Dr. Kerry. Dr. Kerry is conducting research on software security and has collected data from various software systems. Jamal uses Dr. Kerry’s data to investigate a research question he formulated independently, focusing on software vulnerability analysis rather than general software security. His question is his own idea, but is still based on his understanding of software security gained through Dr. Kerry. Jamal’s friend, Darcie, helped Jamal design a specialized software tool for vulnerability detection, but did not contribute in any other way to the research. When writing up his results, Dr. Kerry helped Jamal write the methods section of his manuscript and reviewed his final results and conclusions, as well as the final draft of the entire manuscript"
Ref: University of Minnesota Center for Bioethics. "A Guide to Research Ethics." (2003)
Jamal, Dr. Kerry, Darcie
What is the ethical concercs in the following case?
Students are conducting a study on user behavior in a mobile app. They want to collect user data, including personal information and usage patterns, without explicitly informing the users.
Lack of informed consent
Privacy violation
How would you address the ethical concerns in the following case?
Students are conducting a study on user behavior in a mobile app. They want to collect user data, including personal information and usage patterns, without explicitly informing the users.
Obtain informed consent from users
Inform users about the data collection and usage
Allow users to opt out of data collection
Anonymize the data
Which of the following are examples of plagiarism?
a. quoting a sentence verbatim and citing the author with page number
b. copying and changing a few words in a paragraph without citing the author
c. borrowing an idea from another author and citing them
d. paraphrasing your own previous work with a citation of yourself
b
In the following case, is there any plagiarism and why?
Belinda is publishing her first article that builds on the research of a similar project she did three years prior with her colleague, Isaiah. In Belinda’scurrent article she has placed a graph from the article she and Isaiah co-authored about their previous research. Isaiah created the original graph.
Ref: University of Minnesota Center for Bioethics. "A Guide to Research Ethics." (2003)
Yes, this is plagiarism
Yes, as the graphs have been previously published
Yes, as the idea originated with another entity
What is the ethical concercs in the following case?
Students are conducting a user study to evaluate a new software application. They intentionally select participants who are likely to give positive feedback to validate theirhypothesis.
Bias in participant selection
Manipulation of results
How would you address the ethical concerns in the following case?
Students are conducting a user study to evaluate a new software application. They intentionally select participants who are likely to give positive feedback to validate theirhypothesis.
Random sampling
Objective reporting of results
What is the main ethical concercs in the following case?
A game development student, Alex, is conducting research on player engagement in online multiplayer games. At the same time, Alex's professor, Dr. Harris, serves as a paid consultant for a major gaming company that develops and publishes online multiplayer games.
Conflict of interest
What ethical concerns whould need to be adressed in the following case?
A game development student, Alex, is conducting research on player engagement in online multiplayer games. At the same time, Alex's professor, Dr. Harris, serves as a paid consultant for a major gaming company that develops and publishes online multiplayer games.
Bias in research guidance
Falsification
How would you address the ethical concerns in the following case?
A game development student, Alex, is conducting research on player engagement in online multiplayer games. At the same time, Alex's professor, Dr. Harris, serves as a paid consultant for a major gaming company that develops and publishes online multiplayer games.
Full disclosure
Seek independent review
Separation of roles
Prioritize ethical considerations
What should be done in this case?
Marcus and Clay are game developers collaborating on a research project focused on user feedback for a new mobile game they've created. Clay is responsible for conducting user interviews to gather valuable insights on gameplay and user experiences. One day, Marcus learns that while Clay is interviewing research participants, if he does not elicit an answer, he invents one and passes it off as truthful data collection. Marcus questions Clay and he denies the allegation.
Marcus is obligated to report Clay's activity to the person in charge of the research project
Marcus must continue to go up the chain until with reporting, until action is taken and the behavior stops
Why do we need evidence-based research?
To synthesise the best available evidence, based on multiple studies
Multiple studies might lead in different directions
Aggregate results from multiple studies
What does SLR stand for?
Systematic Literature Review
What is a primary study?
Empirical study in which we directy make measurements about the objects of interests
What is a secondary study?
An analysis of a set of studies, and usually seeks to aggregate the results from these, to provide a stronger forms of evidence
What are forms of secondary studies?
Systematic Literature Review (SLR)
Meta study
Meta analysis
Systematic Literature Mapping
Systematic Mapping Study (SM)
What are key aspects aspects of a systematic literature review?
As described by kitchenham et al. (2007)
A form of a research method
Qualitative or quantitative
What are the phases in a systematic literature review?
As described by kitchenham et al. (2007)
Plan review
Conduct Review
Document review
What are the steps in phase 1 of a systematic literature review?
As described by kitchenham et al. (2007)
Establish need for SLR
Specify RQ
Develop protocol
Validate protocol
What are the steps in phase 2 of a systematic literature review?
As described by kitchenham et al. (2007)
Identify research
Select studies
Assess quality
Extract data
Synthesise
What are the steps in phase 3 of a systematic literature review?
As described by kitchenham et al. (2007)
Document
What are the key factors in step 0, "Establish need for SLR", of a systematic literature review?
As described by kitchenham et al. (2007)
Summarise the existing evidence concerning a treatment or technology
Identify any gaps in current research in order to suggest areas for further investigation
Examine to the extent to which empirical evidence supports / contradicts theorectical hypotheses, or even to assist the generation of new hypotheses
What are the key factors in tep 1, "Specify RQ", of a systematic literature review, with focus on quantitative?
As described by kitchenham et al. (2007)
Comparison of two (or more) technologies to determine which one is more effective or efficient in a certain context.
What are the key factors in tep 1, "Specify RQ", of a systematic literature review, with focus on qualitative?
As described by kitchenham et al. (2007)
Understandingthe benefits, risk, impacts of adopting a specific technology
What does PICOC stand for?
Population: Roles (testers, developers), expertise, industry domain (telecommuncation, automative, etc.)
Intervention: Technology of interests, e.g., TDD, pair programming
Comparison: Test-last development, solo programming
Outcome: Quality (Number of defects, time to complete a task
Context: Academia - industry, SME - Large coporation
What are the key factors in step 2, "Develop protocol", of a systematic literature review?
As described by kitchenham et al. (2007)
Understand the background
Hinges on the research questions
Defines the search strategy
Defines the study selection criteria
Defines the study selection procedues
Defines the study quality assessment checklists and procedures
Defines the data extraction strategy
Defines how synthesis of the extracted data is handled
What are the key factors in tep 3, "Validate protocol", of a systematic literature review?
As described by kitchenham et al. (2007)
Pilot the protocol
Refine the protocol
What are some typical questions you can ask, in regards to refining the protocol in step 3, "Validate protocol", of a systematic literature review?
As described by kitchenham et al. (2007)
Is the motivation clear?
Are the research questions clearly stated?
Is the search strategy clearly defined?
Are the inclusion and exclusion criteria clearly defined?
What are the key factors in step 4, "Identify research", of a systematic literature review?
As described by kitchenham et al. (2007)
Search for relevant primary studies that fulfil the goals of the SLR and answer the research questions
An iterative process
Important to have a basis of completeness
What are the typical search strategies in step 4, "Identify research", of a systematic literature review?
As described by kitchenham et al. (2007)
Automated search
Manual search
Snowballing
What are some key factors of automated search, when searching for primary studies, in a systematic literature review?
Use digital libraries, e.g. IEEE Xplore, ACM Digital Library, etc.
Use services like scopus and Web of Science
Need to compose search strings
Search string usually consists of keywords and operators related to the research questions
Iterative process of composing and refining search strings
What are some key factors of manual search, when searching for primary studies, in a systematic literature review?
Manually search primary studies from journals and conference proceedings
Identify leading software engineering journals and conferences, examples being: ICSE, IEEE, FSE/ESEC, TSE, SoSym, etc.
What are some key factors of snowballing, when searching for primary studies, in a systematic literature review?
Citation anaylsis
Backward, where you look at the references of known papers
Forward, Where you look at other papers that have cited known papers
Suitable where there are a small number of pappers or in new areas
More precise in identifying relevant studies than automated search
Is a single search strategy for primary studies, in a systematic literature review, enough?
No
What are the key factors in tep 5, "Select studies", of a systematic literature review?
As described by kitchenham et al. (2007)
Select studies to include in the SLR
Define inclusion and exclusion criteria
Using a Kappa analysis on a scale from 0 to 1, with multiple reviewers
Funneling process
What are the key factors in step 6, "Assess quality", of a systematic literature review?
As described by kitchenham et al. (2007)
Determine the extent to which the results of an empirical sudy are valid and free (minimised) from bias
The variation of quality needs to be considered when synthesissing the results
A high quality study should carry a heavier weight when synthesising the results
Quality criteria coudl differ between SLR studies
Quality can be assessed using Rigour - Precision or exactness of the research method used
Quality can be assessed using Relevance - Evaluation of the academic relevance or research impact in industry
Multiple people can do the quality assessment, however either aggregation or consensus is needed for scores
In SLR, when accessing quality, what does Rigour include?
Based on Ivarsson and Gorschek, 2011.
Context described
Study design described
Validity discussed
In SLR, when accessing quality, what does Relevance include?
Based on Ivarsson and Gorschek, 2011.
Subjects
Context
Scale
Research method
What are the key factors in step 7, "Extract data", of a systematic literature review?
As described by kitchenham et al. (2007)
Extract the data needed to address the research questions
Could be numerical data or textual data
Uses tools like spreadsheets, analysis tools (NVivo), etc.
What are the key factors in step 8, "Synthesise", of a systematic literature review?
As described by kitchenham et al. (2007), with respect to the primary studies of a common topic.
Summarise
Integrate
Combine
Compare
What are typical types of combination strategy in step 8, "Synthesise", of a systematic literature review?
As described by kitchenham et al. (2007).
Aggregation (qualitative)
Meta-analysis (quantitative)
Vote-counting (quantitative)
What does uantitative synthesis in step 8, "Synthesise", of a systematic literature review, focus on?
Statistical infromation, like correlation, descriptive statistics, etc.
What does vote counting in synthesis in step 8, "Synthesise", of a systematic literature review, focus on?
Counting (or otherwise summarising) the number of studies that show significant efffect
What does meta-analysis in synthesis in step 8, "Synthesise", of a systematic literature review, focus on?
Statistical methods to analyse data from primary studies that compares two different treatments
What should you be aware of, when documenting the SLR in step 9, "Document", of a systematic literature review?
As described by kitchenham et al. (2007)
The audience
Format of the report
What is a Systematic mapping (SM) study?
Secondary study intended to identify and classify the set of publication on a topic
Give an overview of a research area through classification
Counting contributions in relation to the categories of that classification
What are research questions typical in a Systematic mapping (SM) study?
Aimed at discovering research trends
What are research questions typical in a Systematic literature review (SLR) study?
Aimed at aggregating evidence pertaining to the impact of an intervention
What are quality assesment tools in a Systematic mapping (SM) study?
Nice to have
What are the outcome typically in a Systematic mapping (SM) study?
Classification of primary studies
What are the outcome typically in a Systematic literature review (SLR) study?
Qualitative or quantitative synthesis
What are the steps in a Systematic mapping (SM) study?
As described by Petersen et al. (2008)
Need identification & Scoping -> Aim & RQ
Conduct search -> All papers
Study selection -> Relevant papers
Keywording using abstract (classification) -> Classification scheme
Data extraction & mapping -> Systematic map
Document
What are the key factors of step 1, "Need identification & Scoping", in a Systematic mapping (SM) study?
As described by Petersen et al. (2008)
Aimed at identifying trends and gap in literature
What are the main search strategies step 2, "Aim & RQ", in a Systematic mapping (SM) study?
As described by Petersen et al. (2008)
Manual search
Automated search
Snowballing
What are the key factors of step 3, "Conduct search", in a Systematic mapping (SM) study?
As described by Petersen et al. (2008)
Similar to SLR
Relevant to the focus area and within scope
Within the domain
Language scope
What are some of the ways to find topic-dependenden classifications of step 4, "Classification", in a Systematic mapping (SM) study?
As described by Petersen et al. (2008)
Use keywords use in the abstract or list of keywords mentioned in the primary studies
Use existing classification from ISO standards or SWEBOK
Find other similar mapping studies
What are some of the ways to find topic-independenden classifications of step 4, "Classification", in a Systematic mapping (SM) study?
As described by Petersen et al. (2008)
Venue and research method
Research type (Wieringa et al., 2006): evaluation, solution proposal, validation, etc...
What are som keyfactors of step 5, "Data extraction & mapping", in a Systematic mapping (SM) study?
As described by Petersen et al. (2008)
Data is extraced and mapped to the classification scheme
Visualiaztion such as bubble plots, histograms and tables
What are tertiary studies?
Review of secondary studies
What are some of the motiviation behind doing tertiary studies?
Mapping of existing secondary studies
Aggregation of findings in secondary studies
Evaluation of existing secondary studies
What is grey literature?
Non peer-reviewed publications
Often done, when not extensively enough research is done on the topic
Common in education and nursing
What shades of grey literature are there?
1st: High outlet control & credibility: Books, magazines, government reports, white papers, etc.
2nd: Moderate outlet control & credibility: Annual reports, news articles, presentations, videos, Q&A sites, Wiki articles, etc.
3rd: Low outlet control & credibility: Blogs, emails, tweets, forum posts, etc.
What is multivocal literature review?
SLR of both white and grey literature
What is a replication study?
Repeating a study to verify the results are reproducible
Establish creddibility of scientific studies
Increase the validity of the original study
Generalize original study
What is the difference between reproduce and replicate?
Specifically in the context of replication studies.
Reproduce: Repeating the same study with the same data and methods
Replicate: Repeating the same study with different data and the same analysis methods
What are some key areas to focus on, when formulating a search string for online databases?
Keywords
Spelling (UK vs US)
Broad terms
Narrow terms
Translation
Abbreviations
Synonyms
Homonyms
Classifications
What are some of the material types that you can search for, when formulating a search string for online databases?
Books
Journal articles
Conference proceedings
Review articles
Reports
Patents
PHD theses
Statistical reference works
What are some of the key aspects of random search, when searching for information?
Easy to do
Good for preliminary searches
Gain insight
Expand research vocabulary
formulate better query
What are some of the key aspects of cited reference searching, when searching for information?
One key article can lead to more relevant articles
Understanding of research fields development
Moving forward or backward in time
What are some of the weaknesses of cited reference searching?
High number of citations does not guarantee quality
New publications have a few citations
Self citing groups
What are some of the key aspects of evaluating and documenting, when searching for information?
Scan titles and/or abstracts for relevance
Record all findings, and how they where found
Peer-review status
Who is the publisher?
Up to date with current information
What is empirical research?
Direct observations
Experience from the researchers
Case study
Survey
Collect qualitative data
Collect quantitative data
What are typical measurements in empirical research?
Norminal
Ordinal
Interval
Ratio
What are typical qualitative measurements?
Nominal
Ordinal
What are typical quantitative measurements?
Interval
Ratio
What are typical approaches to qualitative data collection
Sampling: Purposefull selection, not random
Interview and focus groups
Observation
Document analysis
What are interviews?
According to Robson (2011)
A type of data collection where researchers ask question
Participants hopefully answer with usefull questions
Conversation with a purpose
Can be done as: Face-to-face
Can be done as: Video call
Can be done as: Email
What are the types of interviews?
According to Robson (2011)
Fully-structured
Semi-structured
Unstructured
What does a fully-structured interview mean?
According to Robson (2011)
Predetermined questions
Fixed wording of questions
Fixed ordering of questions
What does a semi-structured interview mean?
According to Robson (2011)
Interview guide
Checklist of topics
Non-fixed wording
Non-fixed ordering
What does an unstructured interview mean?
According to Robson (2011)
No predetermined questions
No predetermined order
Conversation like
What are some of the advantages and disadvantages of fully-structured interviews?
Flexible
Adaptable
In-depth
Time-consuming
Possible bias
What are some questions to avoid when doing interviews?
According to Robson (2011)
Long questions
Mutli barraled questions
Including uncommon jargon
Leading questions
Biased questions
What is prompts in interviews?
According to Robson (2011)
Suggest a range of possible answers. (Without bias)
What is probing in interviews?
According to Robson (2011)
Ask interviwee to expand on their answers
Pause before asking follow-up questions
Enquiring glance
Summarize response, to ensure understanding
What is part of the interview flow / Protocol?
Introduction
Warm-up
Main part
Cool-off
Closure
What is a focus group?
Interview that take place in a group context
Usually between 8 and 12 people
Where are focus groups often used and why?
Marketing
Pre-cursor to in-depth study
Complement to other methods
What are some of the advantages of focus groups?
Efficient
Large varirty of perspectives
Individuals can build on each others ideas
Encourage to share views
What are some of the disadvantages of focus groups?
Number of questions possible
Facilitation
Conflicts
Confidentiality
Influence
Bias
Domination of individuals
What is a Homogeneous focus group?
According to Robson (2011)
Common background
Common position
Common experience
What is a heterogeneous focus group?
According to Robson (2011)
Diverse background
Diverse position
Diverse experience
What are the key aspects of having a Homogeneous group in a focus group?
According to Robson (2011)
Idea exchange
Easy Communication
Safe environment
Similar answers
What are the key aspects of having a Heterogeneous group in a focus group?
According to Robson (2011)
Rich discussion
Inspire each other
Power imbalance
Different answers
How many people should facilitate a focus group?
Two - Two people can facilitate, with one asking questions and the other taking notes
What are the key factors of observation?
Commonly used in anthropology
Directly observe a specific group
Observation of specific time frames or tasks
Could be used as a pre-cursor to other data collection
Could be used in conjunction with other data collection
What are some key things to observe in observation?
According to Robson (2011)
Space
Actors
Activities
Object
Acts
Events
Time
Goals
Feelings
What are some observation types?
Complete participant
Observer participant
Participant observer
Complete observer
What does Complete participant mean, in observation?
Researcher participates in the group under observation and conceals who they are
What does Observer participant mean, in observation?
Researcher participates in the group under observation and make their role known
What does Participant observer mean, in observation?
Researcher participates in the group under observation, but mostly acts as an observer
What does Complete observer mean, in observation?
Researcher does not participate in the group, and only observes
What are some of the advantages of observation?
Directness
Minimise decoupling in what is said and what is done
What are some of the disadvantages of observation?
Behavior may change when observed
Granularity of observation
Time spent observing
What types of observations are possible?
Structured
Undstructured
What are some of the biasses in observation?
Selective attention
Selective memory
Interpersonal factors
What types of field notes are their for observations?
Note taking
Event logs
Biosensors
key loggers
What is document analysis?
Analysis of non-confidential documents
Analysis of project summaries
Analysis of official reports
Analysis of training materials
What are document analysis used for?
Complement to interview
Complement to observation
Used as prelimenary study
Triangulate data, to support claims
What is coding?
With focus on qualitative data collection and analysis
Condensate data into labeled chunks
Make easy to analyse units
Which description fits Vivo coding?
Literal coding - The terms used by participants themselves
What desciption fits Descrptive coding?
Topic coding - Summarice qualitative data into a single word as the basic topic
What's the first cycle of coding mainly focusing on?
Extract terms from the data
Extract concepts from the data
Extract phrases from the data
What's the second cycle of coding mainly focusing on?
Identify trends
Identify patterns
Label terms into a single label
What type of coding, is usually done in the second cycle of coding?
Focused coding - Identify the most frequent or significant codes
Axial coding - Identify the most frequent or significat categories
Theoretical coding - Identify the releationships between categies
What is important when doing coding?
With focus on qualitative data collection and analysis
Keep a memo when coding
Document thinking process
Interviewing participants multiple times, rather than just once, is an example of triangulation in qualitative research
False
What is another name for validity in qualitative research?
Trustworthiness
What is the optimal number of questions for an interview protocol?
5-10
For a study exploring how kids engage in play at Montessori elementary school, which would be the most appropriate data collection method?
Observation
What is a key characteristic of qualitative data analysis?
Thematic coding and pattern recognition
In qualitative research, what is the term used for the process of ensuring the accuracy and credibility of findings?
Triangulation
What is the main difference between qualitative and quantitative research?
Qualitative research deals with words and meanings, while quantitative research deals with numbers and statistics.
What are some of the typical quantitative research methods?
Surveys
Experiments
What research design does a survey cover?
Qualitative
Quantitative
What is a survey?
An organized method of obtaining data from participants via written or oral questions
It is a comprehensive system for collecting information
What types of surveys does their exist?
Supervised
Semi-supervised
Unsupervised
What does supervised surveys mean?
Interviwer administers the servey togheter with the participants
Ofte telephone
Can be one-to-one
Can be a group
What does Semi-supervised surveys mean?
Interviewer explains the survey, often through example questions, but let the participant fill out the survey themselves
What does unsupervised surveys mean?
Interviewer are not present
Often web-based
Often paper-based
What are some of the research objectives, when performing a survey?
Explain
Describe
Measure a frequency
Evaluate a characteristic of a population
Discover a factor that affect the characteristics of a population
What should you consider when designing a survey?
Multiple people should be in the creation process
If you should measure descriptive
If you should measure behavioural
If you should measure attitudinal
Allignment with RQ
What are some things that could be included in the survey, to increase motivation?
According to Kitchenham and Pfleeger, 2008
What the purpose of the study is
Why it should be of relevance to them
Why each individual's participation is important
How and why each participant was chosen
How confidentiality will be preserved
Try to offer a benefit
How can you test a survey, before deploying it?
According to Kitchenham and Pfleeger, 2008
Piloting
A focus group
Using representative of your sample population
Using people who might use the results
What are some of the advantages of using a survey?
Quickly gather large amount of data
Cheap
Quick
What are some of the disadvantages of using a survey?
Complex to design
Competetive market
What is an experiment?
A scientific procedure undertaken to make a discovery, test a hypothesis, or demonstrate a known fact
What two types of variables are there in experiments?
Independent
Dependent
What are typical types of experiments in Software engineering?
Quasi-experiments
Controlled experiments
What does Quasi-experiments mean?
An experiment that does not have random assignment
An experiment that does not have a control group
Types of members are equally distributed in groups
What are the steps in the experimental process?
Scoping
Planning
Operation
Analysis and interpretation
Presentation and package
When defining experiment scoping, what are the questions to be answered?
What is studied? (Analyse)
What is the intention? (Purpose)
Which effect is studied? (Quality focus)
Whose view? (Perspective)
Where is the study conducted? (Context)
What are the steps in experiment planning?
Context selection
Hypothesis formulation
Variables selection
Slection of subjects
Choose design type
Instrumentation
Validity evaluation
What are some common principles for experiment design?
Randomization - Pick subjects randomly
Blocking - Eliminate undesired effect of a factor
Balancing - Each treatment is applied equally
What are types of subject assignments to treatments in experiments?
Between subjects
Within subjects
When using "Within subjects" slection of subjects, what is it important to be aware of?
Mixing the ordering
Ordering can affect the results
Is bewtween subjects more statistically powerful than within subjects?
No, but the opposite is true
What's the benefits of using between subjects?
Minimize the learning and transfer across conditions
What's the benefits of using within subjects?
Removes individual variation
How can you make sure, that you can trust the result of the experiment (Validity evaluation)?
Inspect instruments, and ensure no errors
Variables represents the concepts from theory
Carefully select subjects
Choose analysis method carefully
Ensure to acknowledge, that the results are not generalizable
What are common types of quantitative data analysis?
Descriptive statistics
Inferential statistics
What does descriptive statistics do?
Summarize the data
Measure tendency
Measure variability
Measure outliers
Measure skewness
What does inferential statistics do?
Make inferences about a population based on a sample
Test hypothesis
Estimate parameters
Is there a difference in the tests that can be used, based on the normality of the data?
"Normality" refers to normal distribution.
Yes, not all tests can be done, if there is a large skewness in the data
What type of hypothesis testing is there?
Referring to hypothesis testing in experiments
Parametric
Non-parametric
What does parametric test entail in hypothesis testing?
Referring to hypothesis testing in experiments
Assumes normal distribution
Data is interval or ratio
What does non-parametric test entail in hypothesis testing?
Referring to hypothesis testing in experiments
Does not assume normal distribution
Data is ordinal
What does a low p-value mean?
Strong evidence against the null hypothesis
Reject the null hypothesis
Can you accept a hypothesis?
No, you can only reject or fail to reject
What is a Validity threat
Anything that can compromise the outcome of your study
Inappropriate election of statistical test
Incorrect measurement instrument in an experiment/survey
Incorrect formulation of interview/survey questions
Biased interpretation of interview transcripts
What are some typical examples of validity threats in interview or survey questions?
Leading questions
Double-negative
What is wrong with this interview question?
How likely are you to recommend working with the design team?
Nothing
What is wrong with this interview question?
What problems do you have with the design team?
Leading question
What is wrong with this interview question?
How would you rate the clarity of the instructions?
Nothing
What is wrong with this interview question?
Were the instructions not unclear?
Double-negative
Why is it important to discuss validity threats?
To communicate, that there is limitations to any study
To highlight research honesty and integrity
To view that the study is trustworthy
Correctly argue the finding of the results without misleading the reader
What are the categories of validity threats?
Internal
external
Construct
conclusion
Descriptive
Interpretive
What does internal validity threats refer to?
Factor/factors that might affect cause and effect relationships but is/are unknown to the researcher
What would a likely validity threat be in this study?
you would like to investigate the relationship between age and gameplay performance. The results indicated negative correlation between age and gameplay performance, which implies that the gameplay performance decreases as the age increases
Internal
What does external validity threats refer to?
Conerned with the ability to generalize the results
What does construct validity threats refer to?
Concerned with whether we measured or captured, what we intend to in relation to our hypothesis or theory to test
What would a likely validity threat be in this study?
Assume a researcher would like to assess the subject's experience in a programming language, measuring the number of courses at the university in computer science.
Construct
What does conclusion validity threats refer to?
The degree to which conclusions/inferences we draw (e.g. about relationships between variable, or based on qualitative data) are reasonable
What does descriptive validity threats refer to?
Related to factual accuracy of the account/data; that is, the researchers are not making up or distorting the things they observed and it is expected to produce descriptively same accounts/data for the same event or situation
What does interpretive validity threats refer to?
Concerns with wether the inferences/conclusions follow from the account (data), not biased by the researchers during analysis
What are some actiosn that can be done, in order to deal with validity threats?
Prevent
Mitigate
Admit
How would you prevent validity threats?
Ensure instruments are well developed
Ensure instruments are verified for correctness
Piloting the study whenever possible
Data collected is alligned with the theory or research question
Select appropriate analysis method for the data
How would you mitigate validity threats?
Redo the study
Avoid inferential statistics
Discard any invalid answers
Triangulate
What does triangulation refer to?
In regards to validity threats
Taking different angles of a phenomenon, to widen the view and increase precision
What types triangulation are there?
In regards to validity threats
Data triangulation
Observer triangulation
Methodological triangulation
Theory triangulation
How would you admit validity threats?
Discuss
Aknowledge
What are some typical examples of validity threats in systematic literature reviews in software engineering?
Incorrect or incomplete search terms in automatic search
iinadequate search strategy
Restricted time span
Bias in study selection
Subjective interpretation about the extracted data
Subjective quality assessment
Why do we need to know about academic writing?
A project, is only as good as how it is communicated
Reader will focus more on the writing than the content, if it does not follow academic convention
Which of the following sentences, are correct academic writing?
"The checklist was developed from a literature review and interviews with 11 industry practitioners"
What are the sections in an IEEE format research paper or report?
Title
Abstract
Introduction
Background / Related works
Research methodology
Results
Discussion
Conclusion
Appendices
How long is a IEEE format research paper or report?
6-10 pages, including references but excluding appendices
What is key factors of the title section in an IEEE format research paper or report?
Should capture the main topic, its importance, attractive and concise
Should not be more than 12 words
Should state the main results (declarative)
Should focus on the subject but not the result (descriptive)
Should include the subject as a question (interrogative)
What should the abstract section in an IEEE format research paper or report contain?
Topic
Motivation
Aim
Methodology
Results
Conclusion
What types of abstracts are there in an IEEE format research paper or report?
Unstructured
Structured
What does the introduction section in an IEEE format research paper or report contain?
Extended version of the abstract
Introduce topic
Reason by the topic is important or relevant
Describe the research question
Highlight the main contributions
Outline the rest of the report
What does the background / related works section in an IEEE format research paper or report contain?
Motivation of why the research is needed
summary of relevant previous research on the topic
Highlight the results of the previous research
Highlight the gaps in the previous research
What does the research methodology section in an IEEE format research paper or report contain?
How you did the project
Describe methodology used to answer the research questions
Describe methodology used in data collection
Describe methodology used in data analysis
What does the results section in an IEEE format research paper or report contain?
Objective representation of the results
Subsections could be alligned with research questions
Subsections could be alligned with main findingx
Must not include how the data gathering was done
Which of the following sentences, are correct academic writing for the results section in an IEEE format research paper or report?
"Our survey suggets that 75% of our respondents do not practice TDD"
What does the discussion section in an IEEE format research paper or report contain?
Interpretation of the results
The meaning of the results
The relation to previous research
What potential differences that may be between this study and the previous research
Strengths and weaknesses of the study
What are the implications of the results
What does the conclusion section in an IEEE format research paper or report contain?
Summary of the whole work
Remind the reader of the proposed solution
Revisit the research questions
Reflect what the results may imply
Suggests a direction for future research
What are different ways of acknowledging sources in an IEEE format research paper or report?
Direct quotes
Paraphrasing
Summarizing
Why is citation important in academic writing?
Acknowledge or credit the original author
Reader can refer back to original sources
Adds credibility to the work
What are some generel tips for writing in an academic language?
Avoid slang
Do not use contractions
First time use of abbreviation, write the full word and abbreviation in brackets