Publications

On July 5, 2023, New York City began enforcing NYC Local Law 2021/144 (the “Law”), which regulates employers’ use of automated employment decision tools (AEDTs) in screening candidates for employment or employees for promotion within NYC. Although the City published final regulations implementing the Law, we believe there remain several important open questions that the City must answer before employers can fully understand their compliance obligations.

Who are Covered Candidates and Employees?

The Law applies to an employer’s use of an AEDT to “screen candidates for employment or employees for promotion within the city.”

Critically, the regulations narrowly define a “candidate for employment” to mean someone “who has applied for a specific employment position” (emphasis added) by “submitting the necessary information or items in the format required by the employer or employment agency.” Based on this definition of “candidate for employment,” the Law would not apply to the use of AEDTs to identify potential candidates who have not actually applied for a specific job. For example, an employer’s use of technology or a tool – such as a job board – that filters or ranks resumes or profiles of candidates who have merely expressed an interest in a general type of position, but who have not applied for or responded to a specific job, would likely fall outside the scope of the Law.

In addition to “candidates for employment,” as described above, the Law also covers screening “employees for promotion within the city.” The term “promotion” would seem to indicate that the employee would need to be placed into a higher ranking position for the Law to apply. Moreover, the use of the word “employee” would imply that the individual must be a current employee of the employer for the Law to apply. Admittedly, however, the phrase “employee for promotion” is not entirely clear.

Finally, the Law only applies to the use of an AEDT “within the city.” The City’s recently published FAQ Guidance states that use of an AEDT in the city means:

• The job location is an office in NYC, at least part time. OR
• The job is fully remote but the location associated with it is an office in NYC. OR
• The location of the employment agency using the AEDT is NYC or, if the location of the employment agency is outside NYC, one of the bullets above is true.

Bullet 1 is fairly clear that if the job is being performed in an office or other worksite located in NYC, then the Law applies. However, there is no explanation for what it means for the job to be performed “at least part time” in NYC – e.g., minimum number of hours or days, frequency, etc., which leaves an unanswered question.

Bullet 2 indicates that a job being performed remotely outside of NYC could fall within the scope of the Law, but it is unclear what is meant by “the location associated with [the job] is an office in NYC.” This could mean that if the job reports to or takes direction from an NYC office, then the Law would apply, but admittedly this too is not clear or further defined.

Bullet 3 states that if the employment agency using the AEDT is in NYC, then the Law applies. However, it is unclear what it means for the location of the employment agency to be in NYC. For example, what if the employment agency has an office in NYC, but the employee representative of the employment agency who is actually using the AEDT is outside of NYC and the job is outside of NYC? In other words, does the Law apply to an employment agency that merely has an office in NYC? In our view, that would be an unreasonable stretch of the Law’s application, but the answer is not clear based on the FAQ Guidance.

What is a Covered Employment Decision?

Further, the Law only applies insofar as the employer is using the AEDT to make an “employment decision,” which focuses on the concept of “screening.” The regulations state that to “screen” means to make a determination about whether a candidate for hiring or an employee for promotion “should be selected or advanced in the hiring or promotion process.” Importantly, the regulations’ use of the term “advanced” means that the AEDT is covered even if the determination is not the last one and additional rounds of decision-making will be applied by a human or another tool. For instance, the AEDT is likely covered if it winnows 1,000 candidates down to 20 because those 20 candidates have “advanced” in the “process” (and 980 candidates have been determined not to advance).

Note that the Law only covers certain hiring and promotion decisions – determinations regarding an individual’s other terms and conditions of employment, such as termination, compensation, work schedule, and others, would generally fall outside the scope of the Law.

What is a Covered AEDT?

Understanding what constitutes an AEDT is no small feat. The Law defines AEDT as any “computational process, derived from machine learning, statistical modeling, data analytics, or artificial intelligence, that issues simplified output, including a score, classification, or recommendation, that is used to substantially assist or replace discretionary decision making for making employment decisions that impact natural persons” (emphasis added). The regulations then seek to clarify the individual component parts of this definition, as follows.

“Machine learning, statistical modeling, data analytics, or artificial intelligence”

To be covered by the Law, the tool must use some kind of “machine learning, statistical modeling, data analytics, or artificial intelligence.” The regulations define this as a “group of mathematical, computer-based techniques: (i) that generate a prediction, meaning an expected outcome for an observation, such as an assessment of a candidate’s fit or likelihood of success, or that generate a classification, meaning an assignment of an observation to a group, such as categorizations based on skill sets or aptitude; and (ii) for which a computer at least in part identifies the inputs, the relative importance placed on those inputs, and, if applicable, other parameters for the models in order to improve the accuracy of the prediction or classification.”

We question whether a software program would be considered to satisfy criteria (ii) above if the program requires human intervention to improve its success rate or quality. The language in (ii) appears to mean that the software program must, at least in part, automatically improve itself (i.e., it’s the “computer” doing the identification of inputs and other parameters “in order to improve the accuracy of the prediction”).

Of course, practically speaking, there is often little opportunity to know whether a third-party’s software program does or does not use “machine learning” because software vendors are often reluctant (to say the least) to share their proprietary software or coding with customers. Often these software programs reside in “black boxes” that are not known or accessible to customers (such as the employers that use them), and therefore there is simply no way for employers to know whether or not the tools do or do not meet the definition of an AEDT under the Law.

At a minimum employers should directly ask their software vendors whether their programs engage in machine learning and/or are covered by the definition of AEDT under the Law. Better still, employers should seek to obtain from software vendors an indemnity in the event that their software programs are found to be covered under the Law – though whether a software vendor will give such an indemnity is an open question.

The New York Staffing Association (NYSA), for which our firm serves as General Counsel, in its lobbying efforts has requested that the City make available to software vendors some kind of certification that the vendors can utilize to represent to their customers that their tools do not engage in machine learning and/or are otherwise not covered by the Law. The City should then issue guidance indicating that employers may rely upon any such good faith certification from a software vendor since the employer itself has no means of evaluating whether the vendor’s proprietary and confidential software does or does not meet the definitions under the Law, including, for example, whether and the extent to which the program engages in “machine learning.”

“Simplified output”

In order to be covered by the Law, the program must also provide a “simplified output.” The regulations define “simplified output” to mean a “prediction or classification,” which is largely described under prong (i) of the “machine learning” definition, above (i.e., “an expected outcome for an observation, such as an assessment of a candidate’s fit or likelihood of success ….”). The regulations instruct that a simplified output “may take the form of a score (e.g., rating a candidate’s estimated technical skills), tag or categorization (e.g., categorizing a candidate’s resume based on key words, assigning a skill or trait to a candidate), recommendation (e.g., whether a candidate should be given an interview), or ranking (e.g., arranging a list of candidates based on how well their cover letters match the job description).”

“Substantially assist or replace discretionary decision making”

The regulations also define “substantially assist or replace discretionary decision making,” a necessary component of a covered AEDT, to include: “(i) to rely solely on a simplified output (score, tag, classification, ranking, etc.), with no other factors considered; or (ii) to use a simplified output as one of a set of criteria where the simplified output is weighted more than any other criterion in the set; or (iii) to use a simplified output to overrule conclusions derived from other factors including human decision-making.”

At first glance, it may appear that this definition could be helpful to employers who would argue that they did not “solely” use the tool to make an employment decision, nor weigh the tool’s output more than any other factors, nor overrule human decision-making. However, while this might plausibly be true at the latter stages of an employment decision (e.g., whom to hire from a list of 10 candidates), the definition may not be so helpful in the early stages of the screening process, such as if the tool winnows 1,000 resumes down to 20.

How to Comply with the Law?

Once it is determined that an employer is using a tool that constitutes an AEDT for hiring or promotion decisions under the Law, the employer must do the following before using the AEDT:

1. Bias Audit

The Law makes it unlawful for an employer to use an AEDT unless the tool was the subject of a bias audit within the one year prior to the employer’s use of the AEDT.

The bias audit must be conducted by an independent auditor, which the regulations define as someone who is capable of exercising objective and impartial judgment on all issues within the scope of a bias audit. Importantly, an independent auditor cannot be someone who (i) is or was involved in using, developing, or distributing the AEDT; (ii) at any point during the bias audit, has an employment relationship with the employer that is using or seeks to use the AEDT or with a vendor that developed or distributes the AEDT; or (iii) at any point during the bias audit, has a direct financial interest or a material indirect financial interest in the employer that is using or seeks to use the AEDT or in a vendor that developed or distributed the AEDT. The unfortunate result of this narrow “independent auditor” definition is that an employer generally will not be able to conduct its own bias audit.

The regulations include a number of samples of bias audits, which, in short, will include calculations of the selection rate or scoring rate (depending on whether the AEDT issues category classifications or scores), including race/ethnicity categories, sex categories, and intersectional categories, and the impact ratio for each category. Again, the audit itself is to be conducted by an independent auditor.

The bias audit is a test of the AEDT to assess its disparate impact based on race/ethnicity and sex through the use of “historical data” or, if unavailable, “test data.” “Historical data” includes the data collected during an employer’s use of an AEDT to assess candidates for hiring or promotion. However, if there is insufficient historical data available to conduct a “statistically significant” bias audit, the employer may use “test data,” which is broadly (and vaguely) defined by the regulations as “any data that is not historical data.” If a bias audit uses test data, the summary of results of the bias audit (described below) must explain why historical data was not used and describe how the test data used was generated and obtained.

The regulations’ instructions with respect to “historical data” and “test data” are confusing in myriad respects. First, given their size and EEO-1 reporting requirements, many employers simply do not collect the types of demographic information envisioned by the Law, including with respect to race, ethnicity, sex, and “intersectional categories,” and, moreover, there is nothing in the Law that specifically requires employers to begin collecting this information. Second, it is not clear how an employer or independent auditor may determine whether “historical data” is or is not “statistically significant,” although the recently published FAQ Guidance seems to afford wide latitude in making this determination, as long as the statistical deficiency is disclosed with the summary of results. And third, it is not clear to us where an employer or independent auditor may obtain “test data” or how to determine whether such test data is sufficient – or even how, exactly, to use such test data with the AEDT. But, once again, the FAQ Guidance seems to strike a fairly deferential tone with respect to these decisions, stating that the summary of results for bias audits using test data need only explain how such data was sourced or developed.

Yet another key issue is how the regulations address “multi-employer” bias audits. Specifically, the regulations provide that “historical data used to conduct a bias audit may be from one or more employers … that use the AEDT.” At first, this guidance seems like a welcomed opportunity for multiple employers to rely upon a single bias audit, including one commissioned by the vendor of the software. However, the regulations then state that the employer may only rely on a bias audit of an AEDT that uses the historical data of other employers only if that employer provided historical data “from its own use of the AEDT to the independent auditor conducting the bias audit or if such employer … has never used the AEDT.” Once again, this limitation is troubling because many employers simply do not collect the types of historical data envisioned by the Law, thereby potentially limiting an employer’s ability to join on to a multi-employer bias audit. Interestingly, however, is that the regulations do not define how much or what types of historical data an employer must provide in order to be eligible to participate in the multi-employer bias audit.

2. Post the Summary of the Bias Audit Results

Before using an AEDT, employers must post to the “employment section” of their website in a clear and conspicuous manner: (i) a summary of the results of the AEDT bias audit and the date of the most recent audit, which must include the source and explanation of the data used to conduct the bias audit, the number of individuals the AEDT assessed that fall within an unknown category, and the number of candidates, the selection or scoring rates, as applicable, and the impact ratios for all categories; and (ii) the date the employer began using the specific AEDT. This information must remain posted for “at least 6 months after [the employer’s] latest use of the AEDT for an employment decision.”

We expect that most of this information would be obtained from the independent auditor. Moreover, the regulations provide that these posting requirements can be met by using an active hyperlink to a website that contains the required summary of results above, as long as the link is clearly identified as a link to the results of the bias audit.

3. Notice to Candidates and Employees

Employers must also provide candidates for employment or promotion who “resides in the City” with notice of the following at least 10 business days before the use of the AEDT: (a) that an AEDT will be used in connection with the assessment or evaluation of the candidate and instructions on how the candidate may request an alternative selection process or accommodation (although, interestingly, there is no guidance regarding the nature of the accommodation and the regulations state that nothing “requires an employer or employment agency to provide an alternative selection process”)^[1]; and (b) the job qualifications and characteristics that the AEDT will use in the assessment. The regulations state that this notice can be provided (i) in a job posting or via U.S. mail or email; (ii) for a candidate for hiring, on the employer’s website in a clear and conspicuous manner; or (iii) for an employee being considered for promotion, in a written policy or procedure, in each case provided the requisite 10 business days’ advance notice is given.

With respect to accommodations, please note, however, that employers may have obligations under other laws to provide reasonable accommodations for a candidate being evaluated by a tool that utilizes artificial intelligence, which the U.S. Equal Employment Opportunity Commission has specifically addressed in recent guidance that is accessible here.

Moreover, the Law requires that employers must: (i) provide information on the employment section of their website about their AEDT data retention policy, the type of data collected for the AEDT, and the source of the data; or (ii) post instructions on the employment section of their website for how to make a written request for such information, and if a written request is received, provide such information within 30 days of the request. (Please note, however, that the regulations appear to indicate that both (i) and (ii) must be posted to the employer’s website, though it would not make sense for employers to have to post both the required information and instructions on how to request the required information since such information would already be on the website.) This information must not be disclosed where disclosure of the information would violate local, state, or federal law, or interfere with a law enforcement investigation, in which case the employer must provide an explanation to the candidate as to why disclosure would violate such a law or interfere with such an investigation.

What are the Civil Penalties for Non-Compliance?

The Law provides for civil penalties of up to $500 for a first violation and each additional violation occurring on the same day as the first violation, and not less than $500 or more than $1,500 for each subsequent violation. Each day an AEDT is used in violation of the Law (e.g., without having conducted a bias audit or without posting the summary of the results of the bias audit, etc.) constitutes a separate violation, and each failure to comply with the candidate notice requirements constitutes a separate violation. Accordingly, civil penalties could be significant.

What’s Next?

As enforcement of the Law now begins, there remain substantial unknowns and key questions associated with the Law and its regulations, to put it mildly. We will continue to seek additional clarity and positive developments for employers and publish updates with respect to the Law as more helpful guidance or other information becomes available.

[1] Please note, however, that employers may have obligations under other laws to provide reasonable accommodations for a candidate being evaluated by a tool that utilizes artificial intelligence, which the U.S. Equal Employment Opportunity Commission has specifically addressed in recent guidance that is accessible here.

For more information on the topic discussed, contact:

• Janae Cummings  |  cummings@thsh.com  |  212-508-6742
• Jason B. Klimpl  |  klimpl@thsh.com  |  212-508-7529
• Randi B. May  |  may@thsh.com  |  212-702-3167
• Elizabeth E. Schlissel  |  schlissel@thsh.com  |  212-508-6714
• Andrew W. Singer  |  singer@thsh.com  |  212-508-6723
• Stacey A. Usiak  |  usiak@thsh.com  |  212-702-3158
• John Walpole  |  Walpole@thsh.com  |  212-702-3164
• Andrew P. Yacyshyn  |  yacyshyn@thsh.com  |  212-508-6792