Examining the Relationship between the Economic Performance of Technology-Based Small Suppliers and Socially Sustainable Procurement

Abstract

Sustainable public procurement plays an important role in addressing not only environmental but also economic and social issues through government acquisitions from technology-based small suppliers. In this context, the objective of this study is to better understand the holistic public procurement process by assessing the operational efficiency of technology-based small suppliers and associating the economic aspect of public procurement with the social aspect, such as women-owned businesses. To this end, we analyzed U.S. Department of Defense Small Business Innovation Research grantees by combining network data envelopment analysis with bootstrap truncated regression analysis. Drawing on the analysis results, we found that (1) there is heterogeneity in the performance of research and development, network building, and commercialization sub-processes, and (2) there is a positive relationship between the overall performance and women-owned small suppliers who excel particularly in network building. The former implies that small suppliers may have different expertise in the chain of public procurement; the latter suggests that woman entrepreneurs with a business network may be able to outperform their counterparts in the public procurement market.

1. Introduction

Purchasing is a critical process for decision-makers to maintain the competitiveness of their organizations [1]. With the emergence of global and local value chains, it became particularly important from a perspective of supply chain management [2]. However, a great body of studies has focused on purchasing in the private sector, which dwarfs the importance of that in the public sector [3]. According to the Organization for Economic Cooperation and Development (OECD) study [4], industrial nations tend to spend about 12% of their gross domestic product (GDP) in public procurement. In the United States (US), specifically, purchasing in the public sector, as of 2017, accounts for about a quarter of government expenditure [4]. Considering the significant size of public procurement, it is meaningful to contribute to the relevant literature.

Recent extant literature on public procurement has embraced the concept of sustainability, often represented by the triple bottom line [5]. Despite its broad concept including not only environmental but also economic (or financial) and social dimensions, a majority of the sustainable public procurement literature has focused on green procurement that concerns purchasing environmentally-friendly products or services [6]. A paucity of literature has investigated the nexus of the economic and social dimensions of sustainable public procurement. In this vein, our study of small suppliers owned by minorities or women is meaningful in that small suppliers contribute significantly to the national economy by creating the majority of jobs and, particularly, socioeconomically disadvantageous small suppliers are closely related to the diversity, equity, and inclusivity issues (the social dimension of sustainable procurement) in the U.S. [7].

From a methodological lens, it is notable that most procurement studies tend to take a qualitative approach, particularly based on interviews with stakeholders in purchasing processes [8]. One of the probable reasons is the data availability and quality issue [9]. Purchasing data may not be easily accessible from outside of organizations. Even within the organizations, they are not readily sharable. Because of that, relatively fewer studies took a quantitative approach. Of them, data envelopment analysis (DEA) has been employed to better understand procurement processes [10]. It has proved its special usefulness in screening suppliers by measuring their efficiency or performance based on multiple input and output factors. Moreover, there are some gaps in the existing literature that explore determinants associated with technology-based small suppliers’ performance (e.g., operational or technical efficiency) in a more systemic and robust manner.

To address the aforementioned issues, we shed more light on the economic and social aspects of sustainable public procurement by looking into companies’ procurement processes through the combination of network DEA (N-DEA, a nonparametric technique) and bootstrap truncated regression analysis (B-TRA, a parametric technique). Through a two-stage analytic framework, we first assessed the public procurement performance of suppliers (economic aspects) and then associated the performance with multiple factors indicating socially sustainable purchasing (social aspects). At the first stage, particularly, we decomposed the public procurement process into three different sub-processes: research and development (R&D), network building, and commercialization. At the second stage, we used B-TRA instead of Tobit regression analysis that has been widely used in previous studies. As shown in Simar and Wilson’s study [11], the former generates more robust estimations (i.e., unbiased coefficients) than the latter does by addressing statistical issues stemming from the finite sample.

To solve data availability concerns and deal with the economic and social dimensions of sustainable public procurement issues, we looked into the U.S. Department of Defense (DoD) Small Business Innovation Research (SBIR) program. The program aims to promote technical innovations and efficiency (economic aspects) and facilitate participation from marginalized populations such as women- or minority-owned small businesses (social aspects) [12]. Particularly, the DoD SBIR program seeks to procure R&D results for national security purpose [13]. Additionally, the SBIR awards data are publicly available. Moreover, the previous studies tended to look into the DoD SBIR program from technological innovation or public venture perspectives, rather than the public procurement lens and by solving relevant issues through qualitative approaches. However, this study seeks to provide a new perspective of the DoD SBIR program by focusing on the sustainable public procurement aspects and utilizing rigorous quantitative approaches.

The remaining sections of this research are organized as follows: Section 2 surveys the literature on the DoD SBIR program and sustainable procurement. Section 3 describes the data, conceptual framework, and methodology with a focus on the N-DEA and B-TRA approaches. Section 4 demonstrates the results of quantitative analyses. Section 5 interprets research outcomes and discusses policy implications. Lastly, Section 6 concludes this study with a summary, limitations, and future research.

2. Study Context

This section describes the context of this study with a focus on the SBIR Program, particularly operated by the DoD. The SBIR program started in the mid-1980s as a public venture to capitalize on the technical capacity of small businesses in the attempt to regain the U.S. technological and economic leadership [14]. Evidenced by a series of successful reauthorizations, the SBIR program has contributed to the national competitiveness by achieving its four major goals: (1) stimulating technological innovation; (2) using small businesses to meet federal R&D needs; (3) fostering and encouraging participation by socially and economically disadvantaged small business concerns (including women-owned ones) in technological innovation; and (4) increasing private-sector commercialization of innovations derived from federal R&D [15]. In particular, as the private sector’s capacity surpasses the public sector’s in some technical areas (e.g., information and communication technology and biotechnology), the SBIR program functions as a conduit for transitioning the state-of-the-art technologies from technology-based small businesses to the DoD [13].

According to the Code of Federal Regulations (specifically, 13 CFR 121.702), the SBIR awardees, as suppliers to the federal agencies, need to meet the following criteria: (1) organized for profit, with a place of business located in the United States; (2) more than 50% owned and controlled by one or more individuals who are citizens or permanent resident aliens of the United States, or by other small business concerns that are each more than 50% owned and controlled by one or more individuals who are citizens or permanent resident aliens of the United States; and (3) no more than 500 employees, including affiliates. In terms of their demographics, only 24% of 2889 SBIR awardees, which have won SBIR Phase II awards from the DoD over the period of 2001 to 2010, were owned by either minorities, women, or veterans [13].

With the passage of multiple reauthorization acts, the SBIR program has been extended in terms of size and coverage [14]. The number of SBIR-participating agencies and the amount of their set-aside budgets both have increased over time. Currently, all federal agencies with a considerable R&D function (specifically, those who have more than USD 100 million of extramural R&D budget) are slated to take part in the SBIR program. Along with its sister program, Small Business Technology Transfer (STTR), the annual total budget of the SBIR program is greater than USD 2 billion. Of them, the DoD is responsible for about the half of total budget, followed by the Department of Health and Human Services (e.g., National Institutes of Health), Department of Energy, National Science Foundation, and National Aeronautics and Space Administration. Within the DoD, Air Force, Navy, and Army, as three major services, represent approximately 32%, 23%, and 18%, respectively, while all other components (e.g., Defense Advanced Research Projects Agency, Missile Defense Agency, and Chemical Biological Defense Program) account for the remaining 27% [13].

Although the overall program is harmonized by the Small Business Administration (SBA), the program is independently operated by each participating agency [16]. As such, each agency seeks to achieve its own objectives in addition to the aforementioned four common main goals. In particular, the DoD makes contracts with small businesses with an intent to procure technologies generated through its SBIR program while other federal agencies provide grants to SBIR awardees along with more research-oriented goals. In addition, DoD components and their laboratories (e.g., Army, Naval and Air Force Research Labs) take extensive measures to generate SBIR topics, encourage women- or minority-owned businesses to apply for their programs, assist selected firms in developing their new ideas and building entrepreneurial networks, and provide additional funds (e.g., Commercialization Readiness Program) to bridge the “valley of death” issue [17].

Drawing on the significant program budget and its contribution to the national competitiveness, the SBIR program has been assessed occasionally by chartered organizations such as the National Academies of Sciences, Engineering, and Medicine (NASEM) and RAND Corporation (see [17,18]) and has been studied by some scholars (see [19,20,21]). However, many of their viewpoints were based on the evaluation of a public venture program rather than public procurement. Thus, their foci were on the growth of SBIR awardees in the private market. Moreover, our study’s focus is on the public procurement market where government agencies act as buyers. For instance, the DoD is a primary purchaser (sometimes only one buyer in a monopsony market) in the market of weapons and munitions (e.g., fighter jets). Additionally, their approaches tend to rely on qualitative methods such as surveys of SBIR awardees, interviews with the SBIR program officers, and case studies of selected companies. Even within quantitative studies, a great body used parametric techniques only. Moreover, our study combines two different types of quantitative methods (parametric and nonparametric) to draw out more rigorous research results.

3. Literature Review

3.1. Small Business in Public Procurement

While there is a belief that small businesses play a pivotal role in the national economy and they need government support programs for entering the public procurement market (e.g., SBIR and 8(a) certification), the market has been dominated by large companies [22]. According to the U.S. General Services Administration’s “Top 100 Contractors Report” in 2019, the amount of top 10 contractors as of 2019 was over USD 170 billion (about 29% of total federal contracting), which was greater than the amount of total small contractors (about 27% of total federal contracting) [23]. In the DoD’s procurement, specifically, large businesses such as Boeing and Lockheed Martin accounted for three-fourths of total contracting, which dwarfed small businesses’ contributions to DoD procurement [24].

Although federal agencies, on occasion, have met their small business contracting goals, they have failed to meet their goals for disadvantaged small businesses [25]. For instance, a five percent goal for women-owned small businesses has not been fulfilled until 2015 since the goal was set in 1994 by the Federal Acquisition Streamlining Act, conceptualized in 2000 by the Equity in Contracting for Women Act, and materialized in 2011 by the Women-Owned Small Business Federal Contract Program [26]. According to

Table 1. Government-wide and DoD’s small business contracting achievement in 2019.

Small Business Contracting Achievement		Small Business	Women-Owned Small Business	SMALL Disadvantaged Business	Service Disabled Veteran-Owned Small Business	HUBZone Located Small Business
Government-wide	Prime contracting	26.50%	5.19%	10.29%	4.39%	2.28%
	Subcontracting	33.27%	5.25%	4.17%	1.95%	1.37%
DoD	Prime contracting	24.16%	4.25%	8.56%	3.25%	1.88%
	Subcontracting	38.60%	5.20%	4.00%	2.10%	1.60%

Source: U.S. Small Business Administration [23,24].

, while the government-wide goal was achieved in 2019, the DoD’s prime contracting did not reach 5% yet. DoD’s prime contracting with service-disabled-veteran-owned or HUBZone-located small businesses is smaller than that with women-owned ones.

3.2. Sustainable Procurement and Data Envelopment Analysis Applications

A great body of procurement studies was focused on supply chain management in the private sector, while a relatively small portion of extant literature investigated public procurement [27]. Regardless of sectoral differences, sustainable procurement has emerged as an important subject since the environmental and social aspects of procurement became one of the criteria in selecting suppliers [28]. Particularly, the public sector has been asked to enhance economic efficiency while alleviating environmental and social footprints through sustainable procurement practices. In developing countries, for instance, ethical (e.g., bribery), safety (e.g., exposure to hazards), and human rights issues (e.g., working conditions) attracted attention from scholars [27]. In the developed countries, more emphasis was placed on the environment and diversity. In the US, particularly, social equity issues over women- or minority-owned suppliers have been critical [29].

To investigate relevant research questions, various research methods have been employed. Of them, most quantitative studies sought to find a better way to evaluate, rank, and select better suppliers. To that end, various techniques, represented by multi-criteria decision-making (MCDM) methods such as the Analytic Hierarchy Process (AHP) and Artificial Neural Network (ANN), have been proposed. For instance, Stević et al. [30] ranked sustainable suppliers in the healthcare industry in Bosnia and Herzegovina by employing Measurement of Alternatives and Ranking according to COmpromise Solution (MARCOS). Milosavljević et al. [31] used Composite I-Distance Indicator (CIDI) to assess the public procurement performance of 30 European countries.

Another stream of analytic methods is DEA, which can measure the efficiency or performance of decision-making units (DMUs; suppliers in this case) based on multiple input and output factors (as criteria).

Table 2. Applications of DEA to sustainable procurement.

Author(s)	Method	Summary	Factors
Niewerth et al. [32]	DEA	This study analyzed the performance of construction tenders in the European Union.	Input: life-cycle costs; construction time Output: environmental concept
Yu and Su [33]	Fuzzy DEA	This study examined the performance of Taiwanese sustainable suppliers in the information and communication industry.	Input: production costs; lead time; supply chain carbon footprints Output: quality; demand quantity
Amindoust [34]	Fuzzy DEA	This study assessed the performance of sustainable suppliers in the automotive parts industry in the Middle East.	Criterion: quality; delivery; technology level; after-sales services; environmental management system; pollution control; work safety and labor health; ethics
Ghoushchi et al. [35]	DEA with imprecise data	This study explored the performance of Iranian sustainable suppliers in the petrochemical industry.	Input: total cost of shipments; the number of shipments; work safety and labor health costs; supplier reputation; eco-design costs Output: the number of the bills received from the supplier without errors; the number of the shipments to arrive on time; the interests and rights of employees; supplier’s green image; green management system
Nemati et al. [36]	DEA with partial impacts between inputs, good and bad outputs	This study investigated the sustainability performance of Iranian cable suppliers.	Input: eco-design cost; the number of shipments per month; total cost of shipments; cost of work safety and labor health Output: the number of bills without error; the number of on-time delivered goods; the number of sent non-defective parts; the number of sent defective parts
Zarbakhshnia and Jaghdani [37]	Network DEA	This study evaluated the performance of Iranian sustainable suppliers in the plastic packing strap industry.	Input: eco-design costs; logistics costs; the number of tune raw materials; reliability costs Intermediate: hazardous substances; the number of sustainable products; fuel cost; cost of labor healthOutput: the number of occupation opportunities; the number of delivered products; CO2 emissions
Milosavljević et al. [38]	Benefit-of-doubts DEA	This study compared the public procurement efficiency of EU member states.	Input (of technological dimension): high-tech exports; patent application; R&D exports Output: one bidder; no calls for bids; aggregation; award criteria; decision speed; reporting quality
Dotoli et al. [39]	Fuzzy DEA and other multi-criteria decision making techniques	This study ranked the public procurement performance of tenders at the European Institution.	Input: price Output (quality factors): technical (e.g., ergonomics, functionality); certifications (e.g., product quality, production quality); conditions (e.g., warranty, post-sales)

summarizes some recent DEA applications in sustainable procurement. Based on the conventional DEA, for example, Niewerth et al. [32] looked into tenders in the construction industry. Yu and Su [33], Amindoust [34], and Ghoushchi [35] addressed the imprecise data issues by employing fuzzy DEA. Nemati et al. [36] incorporated undesirable (e.g., the number of sent defective parts) as well as desirable output (e.g., the number of on-time delivered goods) in their DEA model. Further, Zarbakhshnia and Jaghdani [37] proposed N-DEA to shed light on the black box of the sustainable procurement process and assessed the performance of suppliers in the plastic packing strap industry. In the realm of public procurement, Milosavljević et al. [38] and Dotoli et al. [39] applied DEA to assessed country- and bidder-level performance, respectively, using some technology-related factors (e.g., patent applications and functionality).

In this vein, we also used DEA to evaluate the performance of small suppliers for the DoD acquisition of goods and services. Unlike previous studies, however, we decomposed the public procurement process into three sub-processes, namely R&D, network building, and commercialization, by considering the unique characteristics of technology-based small suppliers. While most procurement-related DEA studies concentrated on the performance-based selection and ranking stage, we extended our study to the next stage where we examined the statistical relationships between the performance and sustainability-related factors with a focus on social equity.

Taking advantage of a good fit of the SBIR program with our research objectives, we tested the following research hypotheses:

Hypothesis 1.

Small suppliers may have different expertise in the chain of public procurement and thus their performance scores may vary across sub-processes of R&D, network building, and commercialization.

Due to the nature of small businesses that have limited resources, they may focus on one of the three sub-processes (e.g., network building sub-process) rather than all. As a result, the performance of a specific sub-process may be higher or lower than that of others.

Hypothesis 2.

Small suppliers owned by marginalized populations may be placed in a preferred position in the public procurement market, and thus their performance may be higher than their counterparts.

Since there are some policy programs targeted for women- or minority-owned small businesses (e.g., Small Business Administration’s 8(a) certification and DoD’s Mentor-Protégé Program), they may have more opportunities to exploit the government acquisition market.

4. Methodology

4.1. Data

This study kept track of 252 elite DoD SBIR grantees. The firms were awarded SBIR Phase II funding (as a follow-up of Phase I) from the DoD over the period of 2001–2010. Out of 2889 firms that won the DoD SBIR awards during the same period, the 252 firms filed more than 15 patents that meet the criteria of “serial innovators” [40]. Given that half of all SBIR awardees have filed no patent application at all and most of them have filed one single patent application, the 252 firms can be regarded as elite technology-based small suppliers.

To measure the economic performance of those small suppliers, we collected various secondary data related to (1) DoD SBIR awards from the SBA’s SBIR database; (2) federal procurement contracts from the Federal Procurement Data System—Next Generation; (3) SBIR grantees’ demographics from the System for Award Management; and (4) patent data from the U.S. Patent and Trademark Office (USPTO). See Figure 1 for the data sources of all variables.

In terms of data collection, it may be worth noting that there is a time lag between input-related data and output-related data to avoid simultaneity. While the former is based on the year 2010, the latter was collected at the end of 2015. Generally, it takes considerable time to transition technologies from the lab (i.e., R&D stage at SBIR Phase II) to market (i.e., commercialization stage at SBIR Phase III). To select the appropriate time lag, we referred to previous studies (see [16,41]).

Table 3. Descriptive data statistics.

Var	Definition	Obs	Mean	SD	Min	Max
Federal_procure	Action obligation of federal procurement contracts (USD million)	252	98.14	292.765	0.19	2433
No_patents	Number of patent applications	252	49.61	94.773	15	1251
No_connections	Eigenvector centrality in the SBIR funding network	252	0.024	0.012	0.001	0.045
SBIR_awards	Amount of SBIR awards (USD million)	252	5.78	11.207	0.29	103.27
No_employees	Number of employees	252	86.17	109.106	2	480
Tech_distance	Technological distance between suppliers and DoD	252	0.3845	0.208	0.0002	0.8805
Age	Age of firms	252	22.17	14.934	2	122
HUBZone_located	Dummy (0 or 1) whether located in HUBZone 1 or not	252	0.012	0.109	0	1
Urban_located	Dummy (0 or 1) whether located in urban areas 2 or not	252	0.369	0.484	0	1
Minority_owned	Dummy (0 or 1) whether owned by minority	252	0.040	0.196	0	1
Women_owned	Dummy (0 or 1) whether owned by women	252	0.044	0.205	0	1
Veteran_owned	Dummy (0 or 1) whether owned by veteran	234	0.034	0.182	0	1

¹ HUBZone: Historically Underutilized Business Zones. SBA and other federal agencies have policy programs to provide preferential access to more federal contracting opportunities to small businesses in the HUBZone; ² urban areas: areas with a population of 50,000 or more following the U.S. Census definition.

shows the descriptive statistics of data used in this study. The first six variables were used for N-DEA as input and output factors, while the last six variables were used for B-TRA as independent variables. The former is explained in more detail in the next subsection. The latter includes factors related to social sustainability, such as HUBZone_located, Urban_located (geospatial dimensions), and Minority_owned, Women_owned, Veteran_owned (ethnic/gender dimensions), as well as Age (control variable).

4.2. Variables and Proxies

Input and output factors, which were included in the N-DEA of the overall public procurement process, are described in more detail.

SBIR awards. The amount of SBIR awards is positively related to the number of patent filings [13]. As a public venture capital, the SBIR program provides a substantial amount of money to technology-based small suppliers. Generally, the program offers USD 150 thousand for Phase I grantees to assess technical feasibility and USD 1 million for Phase II grantees to carry out R&D [42]. Additionally, the DoD SBIR program provides Phase II+ funding to facilitate technology commercialization [17]. These financial resources are critical for technology-based small suppliers to secure funding for materializing their new ideas.

Number of employees. Talents with not only technical/commercial knowledge but also interpersonal skills are essential for R&D as well as network building [43]. Particularly, because valuable scientists or engineers contribute to firms’ specialized knowledge stocks, human resources play a pivotal role in the competitiveness of technology-based companies [44]. In addition, high-quality human resources can develop firms’ social capital by building and broadening their entrepreneurial networks that may be a conduit for financial resources, information, and other resources [45].

Technological distance. Technology-based collaborations (e.g., strategic alliances and joint ventures) tend to take place to fill the gap by supplementing complementary assets [46]. This may apply to the DoD in need of meeting warfighters’ demands that cannot be addressed with in-house capacity but can be solved externally. TBSBs with that capacity can be a solution to the DoD and be placed in an advantageous position in building networks with the DoD. In this regard, technological distance means how dissimilar technologies TBSBs have relative to the DoD. Following Choi and Yeniyurt [47], we calculate the technological distance (TD) using the following formula:

$$T{D}_{ij}=1-\frac{F_i{F}_j^{’}}{{\left[\left(F_i{F}_i^{’}\right)\left(F_j{F}_j^{’}\right)\right]}^{1/2}}$$

(1)

where TD_ij = technological distance; F_i = vector of DoD’s patent portfolio (i.e., distribution of patent applications across patent classes); F_j = vector of small supplier j’s patent portfolio.

Number of patents. Results of R&D usually lead to the filing of patents because organizations want to protect their novel and non-obvious ideas with industrial utility and to recoup their R&D investment through intellectual property rights [48]. Thus, the number of patents (granted patents or patent applications) is widely used as an indicator for technological innovations. Specifically, this is true for small suppliers that seek to obtain external funding because filing more patents enables them to display their technological strength and attract investors [49].

Number of connections. Firms’ social capital may be manifested in the number of ties they have generated [50]. In the military technology market, particularly, connections with the DoD are critical in that the market is characterized by monopsony (i.e., the DoD is a single buyer in the market) [51]. However, all DoD components do not have equal capabilities to procure private-sector technologies; they may vary with their size. For instance, three services (i.e., Air Force, Army, and Navy) may have stronger purchasing power than other relatively small components (e.g., MDA and CBD). In this vein, we use the eigenvector centrality in the SBIR funding network rather than just the degree centrality [52]. Since the funding network is bipartite (i.e., connections between a group of small suppliers and a list of federal agencies without connections between small suppliers and between federal agencies), small suppliers with stronger links to more influential procurers (e.g., three services) may outperform their counterparts. In Figure 2, for instance, 3 Phoenix, Inc. has connections to both Navy and Air Force while 1st Detect Corp. has a connection only to the Office for Chemical and Biological Defense.

Federal procurement contracts. The final output of the overall public procurement is acquisition contracts such as delivery orders made by the DoD in the context of our study [18]. Small suppliers may be able to increase the number of contracts by developing more attractive technologies (represented by the number of patents) and/or by building wider and stronger networks with large DoD components (represented by the number of connections).

4.3. Two-Stage Analytic Framework

To test the first hypothesis, we dissected the process of the technology-oriented public procurement process into three sub-processes: R&D, network building, and commercialization. Most existing literature on public procurement, which applied DEA, used a single-process model by regarding the procurement process as a big black box. To take a more holistic viewpoint, we referred to some previous innovation studies that attempted to shed light on the black box by dividing the whole innovation process into two sub-processes such as R&D and commercialization sub-processes [53,54]. While those studies may work well with well-established large companies, their approach may not apply to relatively nascent small firms that have limited financial, human, and social capital. In start-ups, for instance, workers should have multi-tasking capacity (e.g., working for R&D and network building tasks simultaneously) [55]. In terms of financing, they tend to start with bootstrapping (i.e., minimal funding from personal savings or assistance from family, friend, and other acquaintances), and seek public venture programs (e.g., SBIR) before securing sufficient funding from private equity (e.g., Angels and venture capitals) and crowdfunding sources [56]. Further, some studies confirm that the receipt of SBIR funding provides a positive signal (in terms of technical excellence and market potential) to private-sector funders [57].

Given that the extant literature still does not fully reflect the complex innovation process initiated by small suppliers, we added the network building sub-process in tandem with the R&D sub-process, both of which are followed by the commercialization sub-process. That way, we can bridge the gap in the existing DEA literature that misses the role of social capital in a small business context. Although many studies have underlined the importance of entrepreneurial social networks for the better performance of small businesses, there is little literature that incorporates the network building component into the innovation process in the realm of DEA. As shown in Figure 3, specifically, we included the “Network Building Sub-Process” into the overall technology-based public procurement process.

To address our research hypotheses, we employed a two-stage approach: N-DEA at the first stage, and B-TRA at the second stage. DEA, a nonparametric technique, allows one to measure the performance without making assumptions about the form of the production function based on multiple inputs and outputs. Through network DEA, we further evaluated the economic performance of technology-based small suppliers at three different sub-processes. While a great body of studies (see [58,59]) illustrated the production function of the R&D sub-process (so-called knowledge production function), little literature has looked into the network building and commercialization sub-processes (i.e., network and market generation functions). Particularly, because of the lack of parametric studies on the network and market production functions, we sought to take advantage of DEA’s strength in dealing with any types of production functions.

At a subsequent stage, we applied B-TRA to examine how the economic performance is associated with factors concerning the social aspects of sustainability. To estimate less biased coefficients of parametric models, we applied the bootstrap technique to truncated regression models. Detailed descriptions of N-DEA and B-TRA are presented in Appendix A and Appendix B, respectively.

5. Results

Based on the results of N-DEA,

Table 4. Statistics of efficiency scores.

Efficiency	Obs	Mean	Min	Max
R&D	252	0.3265	0.0155	1
Network building	252	0.2983	0.0133	1
Commercialization	252	0.5701	0.0619	1
Overall	252	0.3983	0.1009	0.9608

summarizes the performance scores at three different sub-processes and the overall score. The commercialization performance score (57%) was the highest, followed by R&D (33%) and network building (30%). The overall performance score was about 40%. More specific results of N-DEA are presented in Appendix C.

To examine the statistical difference between the three different types of efficiency scores, we conducted the Friedman test that is grounded in the nonparametric two-way analysis of variance along with Kendall’s coefficient of concordance [60]. Friedman’s χ² of 115.3988 (p-value = 0.0000) means that the distributions of the three types of efficiency scores are statistically different. Additionally, Kendall’s coefficient of 0.2290 toward zero (against one) indicates little concordance across the efficiency scores.

Figure 4 graphically shows the difference in the distributions of the three efficiency scores and the approximately normal distribution of the overall score. The distributions of R&D and network building efficiency scores show a positive skew while that of commercialization efficiency scores indicate a somewhat negative skew. As a result, overall efficiency demonstrates a symmetrical distribution that looks like a normal distribution with left truncation.

Table 5. Results of truncated regression analyses.

Variable	without Bootstrap				with Bootstrap				Difference
	Model 1a		Model 1b		Model 2a		Model 2b
SBIR_awards †	−0.0323 ***	(−4.46)	−0.0387 ***	(−5.09)	−0.0333 ***	(−4.40)	−0.0391 ***	(−4.95)	−0.0004
No_employees†	−0.0624 ***	(−10.34)	−0.0732 ***	(−9.57)	−0.0634 ***	(−10.07)	−0.0741 ***	(−9.41)	−0.0009
Tech_distance	−0.1237 ***	(−3.85)	−0.1302 ***	(−3.91)	−0.1302 ***	(−3.96)	−0.1364 ***	(−3.98)	−0.0062
No_patents†	−0.0756 ***	(−7.65)	−0.0793 ***	(−7.74)	−0.0845 ***	(−7.60)	−0.0882 ***	(−7.86)	−0.0089
No_connections	2.3012 ***	(3.12)	1.6901 **	(2.05)	2.4362 ***	(3.24)	1.8013 **	(2.15)	0.1112
Federal_procure†			0.0184 ***	(3.16)			0.0184 ***	(3.02)	0.0000
Age†			0.0045	(0.34)			0.0052	(0.38)	0.0007
Urban_located			−0.0152	(−1.09)			−0.0155	(−1.06)	−0.0003
HUBZone_located			0.0449	(0.76)			0.0483	(0.78)	0.0034
Minority_owned			0.0134	(0.39)			0.0117	(0.34)	−0.0017
Women_owned			0.0598 *	(1.94)			0.0589 *	(1.81)	−0.0009
Veteran_owned			−0.0232	(−0.63)			−0.0241	(−0.65)	−0.0009
AIC	−416.80		−392.42		−424.43		−399.46		−7.04
BIC	−392.09		−344.05		−399.73		−351.09		−7.04

^† natural log; *** significant at 1%, ** significant at 5%, * significant at 10%; values in the parenthesis: z score; AIC: Akaike’s information criteria; BIC: Bayesian information criteria.

describes the results of TRAs with and without bootstrap. For the analyses, we used various firm-level variables, particularly related to social sustainability. They included not only demographics (e.g., Age), location (e.g., Urban_located and HUBZone_located), and ownership (e.g., Minority_owned, Women_owned, and Veteran_owned). Depending on the absence or presence of bootstrap and a list of independent variables, Models 1a, 1b, 2a, and 2b were constructed and tested. Models without bootstrap tended to underestimate the absolute value of coefficients when compared to ones with bootstrap. Across all models, input and output factors used in our DEA model were statistically significant. No_connections and Federal_procure were positively associated with performance score, whereas SBIR_awards, No_employees, Tech_distance, and No_patents were negatively associated. One notable point in Models 1b and 2b was that Women_owned was positively associated with performance score while the relationships between performance score and other factors regarding social sustainability were not statistically significant.

To further explore where the significance of Women_owned came from, we also present

Table 6. Coefficients of women-owned suppliers over public procurement process.

Variable	R&D	Network Building	Commercialization	Overall
Women_owned	−0.0056 (−0.16)	0.1287 *** (2.71)	0.0013 (0.04)	0.0598 * (1.94)

*** significant at 1%, * significant at 10%.

where the coefficients of Women_owned at different sub-processes are summarized. Interestingly, Women_owned was statistically significant in relation to the network building sub-process only, which implies that women-owned small suppliers are more likely to be better performers, particularly in building networks with funders.

6. Discussion

Drawing on N-DEA, we first determined three different types of efficiencies (i.e., R&D, network building, and commercialization) of 252 small suppliers who have won DoD SBIR funding. In terms of R&D performance, they are “serial innovators” by meeting the criterion of more than 15 patent applications [40]. That being said, it may be questionable if they are also “serial entrepreneurs” who are successful in the public procurement market by achieving high efficiency in network building and commercialization sub-processes as well as R&D sub-process. As “efficiency-inducing change agents,” they need to optimize the efficiency of knowledge, network, and market productions by managing financial, human, and social capitals better [61].

Relative to efficient performers on the frontier, a majority of firms showed relatively low efficiency scores in R&D and network building (on average 32.65% and 29.83%, respectively) while demonstrating relatively high efficiency scores in commercialization (on average 57.01%). The results support our first research hypothesis. It is consistent with other studies that show a dramatic contrast in efficiency scores at different sub-processes. For instance, Lee et al. [62] assessed the efficiencies of Korean small firms at two sub-processes, such as R&D and commercialization, over the period of 2009–2014, and the average scores were 10.2–14.6% for R&D and 64.9–65.2% for commercialization.

Although it is not directly comparable, the commercialization efficiency of this study was somewhat smaller than those of other studies. In addition to Lee et al. [62], Alvarez and Crespi [63] explored the efficiency of Chilean small manufacturing firms, and the average score was 65%. Grilo and Santos [64] examined the efficiency of Portuguese technology-based start-ups from 2009 to 2011, and the average score was 75.15%. Overall, there is some room for improvement among the U.S. small suppliers to reduce inefficiencies (67.35%, 70.17%, and 42.99% for R&D, network building, and commercialization sub-processes, respectively).

To sustainably involve technology-based small businesses in the federal procurement market, it may need to correct the asymmetry of efficiencies in between R&D/network building and commercialization. The failure to address this issue may lead to negative stigma and risky prejudice against contracting with small businesses [22]. One potential solution would be to render inefficient firms benchmark the efficient ones on the frontier. It suggests some policy programs that seek to facilitate procurement knowledge and experience sharing among the DoD SBIR grantees.

Based on the results of TRAs (see Table 4), only Women_owned was statistically significant of the factors concerned with social sustainability, so our second research hypothesis was partially supported, and we focused on the interpretation of Women_owned. Women_owned had a positive association with overall efficiency, which primarily results from the superiority of women-owned small suppliers in network building. One probable reason is that government agencies have encouraged acquisitions from women-owned firms [65]. For instance, some public policy programs (e.g., SBA’s Women’s Business Centers) have assisted female entrepreneurs in participating in the public procurement market and growing their businesses. Additionally, the DoD has made over USD 230 billion of federal procurement contracts with women- and minority-owned businesses over the period of 2010–2016 [66]. While public policy is helpful for the empowerment of women-owned small suppliers, it does not explain everything in that Minority_owned was not statistically significant although minority-owned small suppliers have also received similar preferences in the public procurement market.

Some other explanations may be available from the network characteristics of women entrepreneurs relative to those of men. McGregor and Tweed [67] compared men- and women-owned small businesses from a networking and mentoring perspective and found that female entrepreneurs with a business network (and a business mentor) outperformed counterparts such as those without a business network or male entrepreneurs. Further, Manello et al. [68] demonstrated that as women in senior roles participated in formal networking activities, the firm’s economic efficiency increased. This reinforces the importance of network building, particularly for women-owned small suppliers. It also implies that public policy toward networking and/or mentoring programs can address gender-based economic and social inequity issues.

Overall, this study contributes to the extant knowledge on the sustainable public procurement domain by adding the following new findings: (a) there is significant room for improvement in network building in the technology-based public procurement market, and addressing this issue may be a critical means not only for the economic aspect but also the social aspect of sustainable procurement; and (b) the economic performance is related to the social dimension (particularly the gender equity issue in this study) of sustainable procurement, and facilitating the participation of women-owned small suppliers in the federal procurement may be an effective way to enhance the overall efficiency of the public procurement process.

7. Conclusions

Sustainable public procurement plays an important role in addressing not only environmental but also economic and social issues through government acquisitions from technology-based small suppliers. In this context, the objective of this study was to better understand the holistic public procurement process by assessing the operational efficiency of technology-based small suppliers and associating the economic aspect of public procurement with the social aspect, such as women-owned businesses. To that end, we analyzed U.S. DoD SBIR grantees by combining network DEA with bootstrap TRA. Drawing on the analysis results, we found that (1) there was heterogeneity in the performance of R&D, network building, and commercialization sub-processes; and (2) there was a positive relationship between the overall performance and women-owned small suppliers who excel particularly in network building. The former implies that small suppliers may have different expertise in the chain of public procurement; the latter suggests that women entrepreneurs with a business network may be able to outperform their counterparts in the public procurement market.

One of the concerns was that the overall performance of U.S. DoD SBIR grantees was relatively low. The overall efficiency based on the average of R&D, network building, and commercialization efficiency scores was about 40%. To improve the overall efficiency, the DoD may need to pay more attention to R&D and network building sub-processes. For example, the DoD would be able to better capitalize on its research laboratories in strengthening the R&D and network building capacity of small suppliers. Additionally, the selection of SBIR awardees needs to be more careful in that those with high potential to grow in a sustainable manner can contribute to efficiency enhancement. Particularly, women-owned small suppliers need to be offered more opportunities to develop their business networks and participate in the public procurement market. In this sense, the DoD needs to reinvigorate the current Mentor–Protégé program and develop new programs aiming at the improvement of sustainable public procurement.

While this study sought to be methodologically rigorous and informative not only to academics but also to practitioners, there are some limitations. First, more studies on the network building sub-process are needed. While a knowledge generation function for the R&D sub-process is well-documented in previous studies, literature on a network generation function is rarely found, particularly in the DEA field. We attempted to develop a network generation function using the number of employees and technological distance as inputs and the number of connections as an output, but it may need more theoretically-grounded justifications. Second, this study may not be generalizable to small suppliers in the traditional industries (e.g., food or construction) in that we focused on the technology-driven public procurement market. The conventional procurement market may have different sets of sub-processes and factors that determine the performance of suppliers. It is hoped that those limitations can be addressed in future studies.