Practicing ArchitecturePracticing Architecture
AEC Research Review: A Knowledge Pyramid Model for Determining Trustworthiness
By Richard L. Hayes, AIA, PhD
The AEC Research Pyramid was developed during the 2012 and 2013 AIA Research Summits, with the premise that the knowledge strata for the academy and practice are connected, but not identical. There are multiple varieties of research within the architectural, engineering and construction (AEC) fields. Individuals and organizations run the full gambit including independent scholars, AEC firms, business consultants and academic institutions. A key to appreciating the research undertaken and reported upon depends on the devotion of the scientific method an individual or organization adopts. Within the AEC sector both qualitative and quantitative research can be done in a way that is consistent with the scientific method. Differences in approach are to be expected, the key is that the application of the scientific method helps to ensure research is conducted in an unbiased way. The research conducted in practice and academy settings depend on different educational contexts, as well as professional experiences. As part of the BRIK initiative, the results from the Research Summits’ AEC Research Pyramid are to be posted in BRIK and expected to assist in clarifying research submissions. (Figure 1. Knowledge Pyramid Model)
Shared knowledge on the academy side is based on rigor. For practice, knowledge is based on applicability. The academy is driven by peer review supporting a tenure track career. Practice is based on usefulness of knowledge for the direct application to service and project delivery. For the profession to have the full advantage of research from both spheres there must be recognition of the value of the overlay and connection of these two bodies of knowledge. It should be noted that as a piece of research “moves up” the pyramid, it does not necessarily increase in applicability, only that the methodology followed more closely resembles the scientific method.
An apparent gap of credibility between the two seems to lie on what was termed “industry review.” On the practice knowledge pyramid, research does not qualify at the level of rigor expected in academic settings. This “industry review” is a product of architects publicly sharing their knowledge within the AEC industry for comment. Research findings in this setting can be relied upon, but practitioners may choose to proceed on the basis of relatively speculative findings.
Twin concepts of reliability and validity must be considered to determine trustworthiness. Reliability is the degree to which an evaluation process yields steady and dependable outcomes. Validity is the quality of being empirically sound. So the underlying unifier for the two sides of the pyramid above is a third side that has to do with achieving trustworthiness, i.e. can you trust the study? This aspect has four levels: D – not reviewed, C – reviewed for structure (grammar, spelling), but opinion not reviewed, B –study opinion reviewed, A – study reviewed for trustworthiness. (Figure 2. Level of Review)
As outlined above, trustworthiness implies some degree of a peer review process. Trustworthiness suggests a peer review process that takes into account CARS. CARS is an evaluation of a source (research) using Credibility, Accuracy, Reasonableness, and Support. The characteristics of these attributes are below.
Author and credentials listed
Well edited in terms of grammar, spelling
Positive well balanced tone
Recently published or considered seminal
Tone or language that implies unbiased attitude
No conflict of interest
Specific points of fact
Source for data or statistics provided or referenced
Documentation provided or referenced
Corroborating sources listed
Well-balanced point of view
Overview of the research pyramid development
The knowledge pyramid model for determining trustworthiness had its genesis during the 2012 AIA Research Summit. During that summit there was discussion on metrics and measurement of AEC research. Out of those discussions the pyramid emerged. The following year’s summit revisited the model with discussions and revisions. Since the conclusion of the 2013 summit, further revisions have occurred. What follows is a brief overview of the summits’ discussion with regard to the pyramid. (Figure 3. From 2012 AIA Research Summit, Pyramid’s first mention)
The major outcomes for this session were items to take into account when establishing metrics/measurements:
Summit participants looked at several options to rank and rate research via a simple matrix diagram. The idea was to establish an easily understood graphic that would help in establishing metrics and measurements. Also the graphic concept was to aid in depicting differences between academy tenure and professional evaluation of research documents. (Figure 4. From 2012 AIA Research Summit, Pyramid Refined)
The idea is to have a regular system where by a piece of research can evaluated by parameters of:
The 2013 Summit revisited the pyramid as a benchmarking tool. A benchmarking model that holds potential for architectural research can be called a “pyramid of inquiry” (POI). The pyramid holds promise, because it combines the matrix concept and the sliding scale. A three sided POI pyramid can represent three realms of research: practice-based, academic-based, and review of rigor. The multiple faces could also be designated for specific undertakings, such as design research, educational research, and forensic studies. In each case the reflection of rigor exists by the tip of the pyramid representing the highest degree of rigor, and statistical validity is assumed. By varying the focus of the pyramid faces, the model becomes scalable and can be applied to the design world.
A refined pyramid model was included in the 2013 Summit Report. By having the pyramid context defined, this enables dialogue allowing the AIA to learn and document value propositions for firms doing research, i.e. what constitutes research for the practitioner? How do researchers help the general practice of architecture?
Figure 2. Level of Review
Figure 3. From 2012 AIA Research Summit, Pyramid’s first mention
Figure 4. From 2012 AIA Research Summit, Pyramid Refined